System and method for data replication using a single master failover protocol

ABSTRACT

A system that implements a data storage service may store data on behalf of storage service clients. The system may maintain data in multiple replicas of various partitions that are stored on respective computing nodes in the system. The system may employ a single master failover protocol, usable when a replica attempts to become the master replica for a replica group of which it is a member. Attempting to become the master replica may include acquiring a lock associated with the replica group, and gathering state information from the other replicas in the group. The state information may indicate whether another replica supports the attempt (in which case it is included in a failover quorum) or stores more recent data or metadata than the replica attempting to become the master (in which case synchronization may be required). If the failover quorum includes enough replicas, the replica may become the master.

This application is a continuation of U.S. patent application Ser. No.15/179,812, filed Jun. 10, 2016, now U.S. patent Ser. No. 10/015,042,which is a continuation of U.S. patent application Ser. No. 14/834,392,filed Aug. 24, 2015, now U.S. Pat. No. 9,367,252, which is acontinuation of U.S. patent application Ser. No. 13/352,326, filed Jan.17, 2012, now U.S. Pat. No. 9,116,862, which are hereby incorporated byreference herein in their entirety.

BACKGROUND

Several leading technology organizations are investing in buildingtechnologies that sell “software-as-a-service”. Such services provideaccess to shared storage (e.g., database systems) and/or computingresources to clients, or subscribers. Within multi-tier e-commercesystems, different resources may be allocated to subscribers and/ortheir applications from whole machines, to CPU, to memory, to networkbandwidth, and to I/O capacity.

Database systems managing large amounts of data on behalf of users maydistribute and/or replicate that data across two or more machines, oftenin different locations, for any of a number of reasons, includingsecurity issues, disaster prevention and recovery issues, data localityand availability issues, etc. These machines may be configured in anynumber of ways, including as a shared resource pool.

Interaction between client applications and database servers typicallyincludes read operations (read-only queries), write operations (to storedata), and update operations that can be conceptualized using aread-modify-write workflow.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram illustrating one embodiment of a system thatprovides various Web-based services to clients.

FIG. 1B is a block diagram illustrating one embodiment of a systemarchitecture that is configured to implement a web services-based datastorage service.

FIGS. 2A-2C are block diagrams illustrating various components of a Webservices platform, according to one embodiment.

FIGS. 3A and 3B are block diagrams illustrating the storing of data asitems in a plurality of tables, according to one embodiment.

FIG. 4 is a flow diagram illustrating one embodiment of a workflow forcreating a table.

FIG. 5 is a flow diagram illustrating one embodiment of a method forpartitioning a table maintained in a non-relational data store.

FIG. 6 is a flow diagram illustrating another embodiment of a method forperforming a query.

FIG. 7 is a block diagram illustrating a portion of a data model for asystem that provides data storage services, according to one embodiment.

FIG. 8 is a flow diagram illustrating one embodiment of a method formoving a replica of a partition of a table being maintained by a datastorage service on behalf of a storage service client while thepartition is “live”.

FIG. 9 is a flow diagram illustrating one embodiment of a method forcopying a replica using a physical copy mechanism.

FIG. 10 is a flow diagram illustrating one embodiment of a method forsplitting a partition of a table being maintained by a data storageservice in response to a request to do so.

FIG. 11 is a flow diagram illustrating one embodiment of a method formoving a partition of a table being maintained by a data storage servicein response to detecting an anomaly.

FIG. 12 illustrates various states in a log sequence while undergoing areplication process, according to one embodiment.

FIG. 13 illustrates a commit flow for a log replication mechanism,according to one embodiment.

FIG. 14 illustrates a data replication flow from the perspective of amaster replica for a replica group, according to one embodiment.

FIG. 15 illustrates a data replication flow from the perspective of aslave replica in a replica group, according to one embodiment.

FIG. 16 is a flow diagram illustrating one embodiment of a method forreplicating a write operation in a data storage system.

FIGS. 17A and 17B are flow diagrams illustrating different embodimentsof a method for performing a read operation in a data storage system.

FIG. 18 is a flow diagram illustrating one embodiment of a method forperforming a replication failover process in a data storage system.

FIG. 19 is a flow diagram illustrating one embodiment of a method foracquiring an external lock for a replica group.

FIG. 20 is a flow diagram illustrating one embodiment of a method forfilling out a failover quorum.

FIG. 21 is a flow diagram illustrating one embodiment of a method forperforming a catch-up operation on a log stream tail.

FIG. 22 is a flow diagram illustrating one embodiment of a method forperforming a replica group membership change.

FIG. 23 is a flow diagram illustrating one embodiment of a method forsynchronizing up to a replica group membership change during failover.

FIG. 24 is a flow diagram illustrating one embodiment of a method forsplitting a replicated partition.

FIG. 25 is a flow diagram illustrating one embodiment of a method forreleasing mastership of a partition when it is split.

FIG. 26 is a flow diagram illustrating one embodiment of a method fordetecting and resolving log conflicts in a data storage system.

FIG. 27 is a flow diagram illustrating another embodiment of a methodfor filling out a failover quorum.

FIG. 28 is a flow diagram illustrating one embodiment of a method foremploying an external service or manager to select a master replica fora replica group.

FIG. 29 is a flow diagram illustrating one embodiment of a method foremploying a heartbeat mechanism between an external service or managerand a master replica for a replica group.

FIG. 30 is a flow diagram illustrating one embodiment of a method forcontinuing to service write operations when an external service ormanager is unavailable.

FIG. 31 is a flow diagram illustrating another embodiment of a methodfor continuing to service write operations when an external service ormanager is unavailable.

FIG. 32 is a flow diagram illustrating one embodiment of a method foremploying a series of local leases to determine the replica authorizedto service consistent read operations.

FIG. 33 is a flow diagram illustrating one embodiment of a method fordetermining the replica authorized to service consistent read operationswhen mastership of a replica group changes.

FIG. 34 is a flow diagram illustrating another embodiment of a methodfor determining the replica authorized to service consistent readoperations when mastership of a replica group changes.

FIG. 35 is a block diagram illustrating a computing node that may besuitable for implementation of a data storage service, according tovarious embodiments.

While embodiments are described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that the embodiments are not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit embodiments tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope as defined by the appended claims. The headings usedherein are for organizational purposes only and are not meant to be usedto limit the scope of the description or the claims. As used throughoutthis application, the word “may” is used in a permissive sense (i.e.,meaning having the potential to), rather than the mandatory sense (i.e.,meaning must). Similarly, the words “include”, “including”, and“includes” mean including, but not limited to.

DETAILED DESCRIPTION OF EMBODIMENTS

The systems and methods described herein may be employed in variouscombinations and in various embodiments to implement a Web-based servicethat provides data storage services to storage service clients (e.g.,user, subscribers, or client applications that access the data storageservice on behalf of users or subscribers). The service may in someembodiments support the seamless scaling of tables that are maintainedon behalf of clients in a non-relational data store, e.g., anon-relational database. The service may provide a high level ofdurability and availability through replication, in some embodiments.

In some embodiments, the service may support automatic liverepartitioning of data in response to the detection of various anomalies(e.g., failure or fault conditions, hot spots, or increases in tablesize and/or service request throughput), and/or explicit (e.g.,pro-active and/or subscriber-initiated) live repartitioning of data tosupport planned or anticipated table size and/or throughput increases.In other words, the service may in some embodiments initiate there-sizing (scaling) and/or repartitioning of a table in response toreceiving one or more requests to store, retrieve, modify, or deleteitems in the scalable table.

The service described herein may in various embodiments support aflexible schema, a plurality of available consistency models, a varietyof service level and/or business model options, multiple indexingoptions, and/or multiple query types. In some embodiments, storageservice clients (e.g., users, subscribers or client applications) mayinteract with the service through a Web service interface using arelatively small (and relatively simple) set of APIs, such that clientsof the service are largely relieved from the burden of databaseadministration. The service may exhibit low latency in servicingrequests. Unlike in some prior data storage services, the service mayoffer predictable performance at a low cost, while supportingmulti-tenancy and automatic heat management.

In various embodiments, the data storage service described herein mayprovide an application programming interface (API) that includes supportfor some or all of the following operations on the data in a tablemaintained by the service on behalf of a storage service client: put (orstore) an item, get (or retrieve) one or more items having a specifiedprimary key, delete an item, update the attributes in a single item,query for items using an index, and scan (e.g., list items) over thewhole table, optionally filtering the items returned. The amount of workrequired to satisfy service requests that specify these operations mayvary depending on the particular operation specified and/or the amountof data that is accessed and/or transferred between the storage systemand the client in order to satisfy the request.

In some embodiments, the service (and/or the underlying system thatimplements the service) may support a strong consistency model, inaddition to supporting eventually consistent read operations. In someembodiments, service requests made via the API may include an indicationof one or more user preferences, such as a preferred consistency model,a preferred service request throughput level, or a service requestthroughput level for which a guarantee is requested. In otherembodiments, some or all of these user preferences may be specified whena table is created, or may be client-specific, account-specific,specific to various table types, or specified by system-wide defaultvalues, rather than being specified on a per-request basis. The API maysupport extreme scaling and/or more predictable performance than thatprovided by prior data storage systems and services.

In some embodiments, the service (and/or the underlying system) mayimpose an upper bound on the size of an individual item, e.g., to allowthe service to store the entire contents of an item in a singlepartition in the underlying data storage system. This may, in turn,facilitate performing atomic updates to an item without dramaticallyreducing throughput, and may make it easier to maintain item contents ina stable working set. In other words, limiting the size of an individualitem may facilitate both strong consistency and high performance in thesystem, in some embodiments.

In various embodiments, systems described herein may store data inreplicated partitions on multiple storage nodes (which may be located inmultiple data centers) and may implement a single master failoverprotocol. In some embodiments, membership in various replica groups maybe adjusted through replicated changes, and membership and other updatesin the system may be synchronized by synchronizing over a quorum ofreplicas in one or more data centers at failover time using a replicatedquorum version. In some embodiments, a mechanism for splitting apartition may utilize failover quorum synchronization, external masterlocks, and/or various methods for detecting and resolving log conflicts,including log snipping (e.g., deleting log records that are on invalidbranches). The systems described herein may implement a fault-tolerantlog shipping based replication mechanism that includes such log conflictdetection and resolution. In some embodiments, log branching may beavoided through post-failover rejoins.

In some embodiments, a data storage system may employ an externalservice or manager (e.g., an external lock service or lock manager) toselect a master replica for a replica group. The master replica mayemploy a quorum based mechanism for performing replicated writeoperations that are directed to the replica group (or a correspondingdata partition stored by the replica group), and a local lease mechanismfor determining the replica authorized to perform consistent readsdirected to the replica group (or corresponding data partition), evenwhen the external service/manager is unavailable. The master replica maypropagate local leases to replica group members as replicated writes. Ifanother replica assumes mastership for the replica group, it may notbegin servicing consistent read operations that are directed to thereplica group until the lease period for a current local lease expires.

Various techniques described herein may be employed in local or remotecomputing systems, including systems that provide services to users(e.g., subscribers) over the Internet or over other public or privatenetworks, such as virtual private networks and connections to servicesin a virtual private cloud (VPC) environment. FIG. 1A illustrates ablock diagram of a system that provides various Web-based services toclients, according to one embodiment. In this example, system 100includes one or more clients 105. In this example, the clients 105 maybe configured to interact with a Web server 135 via a communicationnetwork 115.

As illustrated in this example, the Web server 135 may be configured toprocess requests from clients 105 for various services, such as Webservice A (125), Web service B (145), and Web service C (155), and toreturn results to the clients 105. Each of the web services may provideclients with one or more of: computational resources, database services,data storage services (e.g., maintaining data in one or more tables onbehalf of a client), or any other types of services or shared resources.

One embodiment of a system architecture that is configured to implementa Web services-based data storage service such as that described hereinis illustrated in FIG. 1B. It is noted that where one or more instancesof a given component may exist, reference to that component herein belowmay be made in either the singular or the plural. However, usage ofeither form is not intended to preclude the other. In variousembodiments, the components illustrated in FIG. 1B may be implementeddirectly within computer hardware, as instructions directly orindirectly executable by computer hardware (e.g., a microprocessor orcomputer system), or using a combination of these techniques. Forexample, the components of FIG. 1B may be implemented by a distributedsystem including a number of computing nodes (or simply, nodes), such asthe computer node embodiment illustrated in FIG. 35 and discussed below.In various embodiments, the functionality of a given storage servicesystem component may be implemented by a particular computing node ormay be distributed across several computing nodes. In some embodiments,a given computing node may implement the functionality of more than onestorage service system component.

Generally speaking, storage service clients 110 a-110 n may encompassany type of client configurable to submit web services requests to Webservices platform 130 via network 120. For example, a given storageservice client 110 may include a suitable version of a web browser, or aplug-in module or other type of code module configured to execute as anextension to or within an execution environment provided by a webbrowser to provide storage service clients (e.g., client applications,users, and/or subscribers) access to the data storage services providedby Web services platform 130. Alternatively, a storage service client110 may encompass an application such as a database application, mediaapplication, office application or any other application that may makeuse of persistent storage resources. In some embodiments, such anapplication may include sufficient protocol support (e.g., for asuitable version of Hypertext Transfer Protocol (HTTP)) for generatingand processing web services requests without necessarily implementingfull browser support for all types of web-based data. That is, storageservice client 110 may be an application configured to interact directlywith Web services platform 130. In various embodiments, storage serviceclient 110 may be configured to generate web services requests accordingto a Representational State Transfer (REST)-style web servicesarchitecture, a document- or message-based web services architecture, oranother suitable web services architecture.

In some embodiments, storage service client 110 may be configured toprovide access to web services-based storage to other applications in amanner that is transparent to those applications. For example, storageservice client 110 may be configured to integrate with an operatingsystem or file system to provide storage in accordance with a suitablevariant of the storage model described herein. However, the operatingsystem or file system may present a different storage interface toapplications, such as a conventional file system hierarchy of files,directories and/or folders. In such an embodiment, applications may notneed to be modified to make use of the storage system service modeldescribed herein. Instead, the details of interfacing to Web servicesplatform 130 may be coordinated by storage service client 110 and theoperating system or file system on behalf of applications executingwithin the operating system environment.

Storage service clients 110 may convey web services requests to andreceive responses from Web services platform 130 via network 120. Invarious embodiments, network 120 may encompass any suitable combinationof networking hardware and protocols necessary to establish web-basedcommunications between clients 110 and platform 130. For example,network 120 may generally encompass the various telecommunicationsnetworks and service providers that collectively implement the Internet.Network 120 may also include private networks such as local areanetworks (LANs) or wide area networks (WANs) as well as public orprivate wireless networks. For example, both a given client 110 and Webservices platform 130 may be respectively provisioned within enterpriseshaving their own internal networks. In such an embodiment, network 120may include the hardware (e.g., modems, routers, switches, loadbalancers, proxy servers, etc.) and software (e.g., protocol stacks,accounting software, firewall/security software, etc.) necessary toestablish a networking link between given client 110 and the Internet aswell as between the Internet and Web services platform 130. It is notedthat in some embodiments, storage service clients 110 may communicatewith Web services platform 130 using a private network rather than thepublic Internet. For example, clients 110 may be provisioned within thesame enterprise as the data storage service (and/or the underlyingsystem) described herein. In such a case, clients 110 may communicatewith platform 130 entirely through a private network 120 (e.g., a LAN orWAN that may use Internet-based communication protocols but which is notpublicly accessible).

Generally speaking, Web services platform 130 may be configured toimplement one or more service endpoints configured to receive andprocess web services requests, such as requests to access tablesmaintained on behalf of clients/users by a data storage service, and/orthe items and attributes stored in those tables. For example, Webservices platform 130 may include hardware and/or software configured toimplement various service endpoints and to properly receive and processHTTP-based web services requests directed to those endpoints. In oneembodiment, Web services platform 130 may be implemented as a serversystem configured to receive web services requests from clients 110 andto forward them to various components that collectively implement a datastorage system for processing. In other embodiments, Web servicesplatform 130 may be configured as a number of distinct systems (e.g., ina cluster topology) implementing load balancing and other requestmanagement features configured to dynamically manage large-scale webservices request processing loads.

As illustrated in FIG. 1B, Web services platform 130 may include a frontend module 140 (which may be configured to receive, authenticate, parse,throttle and/or dispatch service requests, among other things), one ormore administrative components, or auto admin instances, 150 (which maybe configured to provide a variety of visibility and/or controlfunctions, as described in more detail herein), and a plurality ofstorage node instances (shown as 160 a-160 n), each of which maymaintain and manage one or more tables on behalf of clients/users or onbehalf of the data storage service (and its underlying system) itself.Some of the functionality provided by each of these types of componentsis described in more detail herein, according to various embodiments.

In various embodiments, Web services platform 130 may be configured tosupport different types of web services requests. For example, in someembodiments, platform 130 may be configured to implement a particularweb services application programming interface (API) that supports avariety of operations on tables that are maintained and managed onbehalf of clients/users by the data storage service system (and/or datastored in those tables). Examples of the operations supported by such anAPI are described in more detail herein.

In addition to functioning as an addressable endpoint for clients' webservices requests, in some embodiments Web services platform 130 mayimplement various client management features. For example, platform 130may coordinate the metering and accounting of client usage of webservices, including storage resources, such as by tracking theidentities of requesting clients 110, the number and/or frequency ofclient requests, the size of tables and/or items stored or retrieved onbehalf of clients 110, overall storage bandwidth used by clients 110,class of storage requested by clients 110, and/or any other measurableclient usage parameter. Platform 130 may also implement financialaccounting and billing systems, or may maintain a database of usage datathat may be queried and processed by external systems for reporting andbilling of client usage activity. In some embodiments, platform 130 mayinclude a lock manager and/or a bootstrap configuration (not shown).

In various embodiments, a data storage service may be implemented on oneor more computing nodes that are configured to perform the functionalitydescribed herein. In some embodiments, the service may be implemented bya Web services platform (such as Web services platform 130 in FIG. 1B)that is made up of multiple computing nodes, each of which may performone or more of the functions described herein. Various collections ofthe computing nodes may be configured to provide the functionality of anauto-admin cluster, a cluster of resources dedicated to the data storageservice, and a collection of external resources (which may be sharedwith other Web services or applications, in some embodiments).

In some embodiments, the external resources with which the systeminteracts to provide the functionality described herein may include anexternal workflow component, illustrated in FIG. 1B as external workflowcomponent 170. External workflow component 170 may provide a frameworkthrough which other components interact with the external workflowsystem. In some embodiments, Web services platform 130 may include anaccess API built on top of that framework (not shown). This interfacemay allow the system to implement APIs suitable for the usage patternsexpected to be experienced by the data storage service. In someembodiments, components or modules of the system that use externalworkflow component 170 may include these interfaces rather thaninterfacing directly to the interfaces provided by external workflowcomponent 170. In some embodiments, the Web services platform 130 mayrely on one or more external resources, such as an external storageservice 180, and/or other external (and in some cases shared) externalresources, in addition to external workflow component 170. In someembodiments, external workflow component 170 may be used to performdistributed operations, such as those that extend beyond a particularpartition replication group.

FIGS. 2A-2C illustrate various elements or modules that may be includedin each of the types of components of Web services platform 130,according to one embodiment. As illustrated in FIG. 2A, front end module140 may include one or more modules configured to perform parsing and/orthrottling of service requests (shown as 210), authentication and/ormetering of service requests (shown as 215), dispatching servicerequests (shown as 225), and/or maintaining a partition map cache (shownas 230). In addition to these component-specific modules, front endmodule 140 may include components that are common to multiple types ofcomputing nodes that collectively implement Web services platform 130,such as a message bus (shown as 235) and/or a dynamic configurationmodule (shown as 240). In other embodiments, more, fewer, or differentelements may be included in front end module 140, or any of the elementsillustrated as being included in front end module 140 may be included inanother component of Web services platform 130 or in a componentconfigured to interact with Web services platform 130 to provide thedata storage services described herein.

As illustrated in FIG. 2B, auto admin instance 150 may include one ormore modules configured to provide visibility and control to systemadministrators (shown as 245), or to perform heat balancing (shown as250), and/or anomaly control (shown as 255), resource allocation (shownas 260). Auto admin instance 150 may also include an admin console 265,through which system administrators may interact with the data storageservice (and/or the underlying system). In some embodiments, adminconsole 265 may be the primary point of visibility and control for thedata storage service (e.g., for configuration or reconfiguration bysystem administrators). For example, admin console 265 may beimplemented as a relatively thin client that provides display andcontrol functionally to system administrators and/or other privilegedusers, and through which system status indicators, metadata, and/oroperating parameters may be observed and/or updated. In addition tothese component-specific modules, auto admin instance 150 may alsoinclude components that are common to the different types of computingnodes that collectively implement Web services platform 130, such as amessage bus (shown as 235) and/or a dynamic configuration module (shownas 240). In other embodiments, more, fewer, or different elements may beincluded in auto admin instance 150, or any of the elements illustratedas being included in auto admin instance 150 may be included in anothercomponent of Web services platform 130 or in a component configured tointeract with Web services platform 130 to provide the data storageservices described herein.

As illustrated in FIG. 2C, storage node instance 160 may include one ormore modules configured to provide partition management (shown as 270),to implement replication and failover processes (shown as 275), and/orto provide an application programming interface (API) to underlyingstorage (shown as 280). As illustrated in this example, each storagenode instance 160 may include a storage engine 285, which may beconfigured to maintain (i.e. to store and manage) one or more tables(and associated table data) in storage 280 (which in some embodimentsmay be a non-relational database) on behalf of one or moreclients/users. In addition to these component-specific modules, storagenode instance 160 may include components that are common to thedifferent types of computing nodes that collectively implement Webservices platform 130, such as a message bus (shown as 235) and/or adynamic configuration module (shown as 240). In other embodiments, more,fewer, or different elements may be included in storage node instance160, or any of the elements illustrated as being included in storagenode instance 160 may be included in another component of Web servicesplatform 130 or in a component configured to interact with Web servicesplatform 130 to provide the data storage services described herein.

The systems underlying the data storage service described herein maystore data on behalf of storage service clients (e.g., clientapplications, users, and/or subscribers) in tables containing items thathave one or more attributes. In some embodiments, the data storageservice may present clients/users with a data model in which each tablemaintained on behalf of a client/user contains one or more items, andeach item includes a collection of attributes. The attributes of an itemmay be a collection of name-value pairs, in any order. In someembodiments, each attribute in an item may have a name, a type, and avalue. Some attributes may be single valued, such that the attributename is mapped to a single value, while others may be multi-value, suchthat the attribute name is mapped to two or more values. In someembodiments, the name of an attribute may always be a string, but itsvalue may be a string, number, string set, or number set. The followingare all examples of attributes: “ImageID”=1, “Title”=“flower”,“Tags”={“flower”, “jasmine”, “white”}, “Ratings”={3, 4, 2}. The itemsmay be managed by assigning each item a primary key value (which mayinclude one or more attribute values), and this primary key value mayalso be used to uniquely identify the item. In some embodiments, a largenumber of attributes may be defined across the items in a table, buteach item may contain a sparse set of these attributes (with theparticular attributes specified for one item being unrelated to theattributes of another item in the same table), and all of the attributesmay be optional except for the primary key attribute(s). In other words,unlike in traditional databases, the tables maintained by the datastorage service (and the underlying storage system) may have nopre-defined schema other than their reliance on the primary key. Notethat in some embodiments, if an attribute is included in an item, itsvalue cannot be null or empty (e.g., attribute names and values cannotbe empty strings), and, and within a single item, the names of itsattributes may be unique.

In some embodiments, the systems described herein may employ a somewhatlimited indexing and/or query model in order to provide massive (i.e.virtually unlimited) scaling, predictability, and simplicity forusers/subscribers or client applications. For example, in someembodiments, data may be indexed and partitioned (e.g., partitioned inthe underlying database) by a primary key only. In such embodiments, theprimary key to be used for indexing data in a user table may bespecified by the user at the time that the table is created on theuser's behalf. Thereafter, the partitioning of the user's data may behandled by the system, and abstracted from the user. In someembodiments, the primary key used for indexing data may consist of asingle attribute hash key. In other embodiments, the primary key usedfor indexing and/or partitioning data may be a composite key comprisinga hash key component and another component, sometimes referred to hereinas a range key component. In various embodiments, queries may besupported against indexed attributes, and a full table scan function maybe provided (e.g., to support troubleshooting). In some embodiments,users may define secondary indexes for a table based on one or moreattributes other than those of the primary key, and then may query foritems using the indexes they have defined. For example, in someembodiments the system may support the creation of creating secondaryindexes on-the-fly (e.g., using a createIndex API), and these secondaryindexes may scale automatically based on storage requirements (e.g.,increasing or decreasing data volume) and/or read/write traffic. In someembodiments, such secondary indexes may be asynchronously updated asitems in the table are updated.

In various embodiments, the service (and/or the underlying system) mayenforce pre-determined size limits on table names, items, attributevalues, primary key values, and/or attribute names. For example, in someembodiments, the total size of all the attribute names and values in anitem (i.e. the row size) may be limited.

FIGS. 3A and 3B illustrate the storing of data in a plurality of tables,according to one embodiment. As illustrated in FIG. 3A and describedabove, each of a plurality of tables (shown as tables 320 a-320 n) maystore a plurality of items. In the illustrated example, table 320 astores items 321 a-321 n, and table 320 n stores items 322 a-322 n. Asillustrated in FIG. 3B, each of the items stored in a table may includea plurality of attributes, and each of the attributes may include anattribute name and a scalar or set type value. In this example, item 321a (stored in table 320 a) includes a numeric “imageID” attribute whosevalue is 1, a numeric “date” attribute whose value is 20100915, a stingattribute named “title” whose value is “flower”, and a string attributenamed “tags” whose value is the set containing the strings “flower”,“jasmine”, and “white”. In this example, item 321 b (which is alsostored in table 320 a) includes a numeric “imageID” attribute whosevalue is 2, a numeric attribute named “ratings” whose value is the setcontaining the numeric values 3, 4, and 2, a string attribute named“title” whose value is “credenza”, a numeric “width” attribute whosevalue is 1024, and a numeric “depth” attribute whose value is 768. Inthis example, item 321 n (which is also stored in table 320 a) includesa numeric “imageID” attribute whose value is n, a numeric “date”attribute whose value is 20110327, and a string attribute named “tags”whose value is the set containing the strings “france” and“architecture”. Note that even though items 321 a, 321 b, and 321 n areall stored in the same table (table 320 a), they do not all include thesame set of attributes. Instead, each item includes a sparse set ofattributes from among all the attributes that have been specified forthe collection of items stored in table 320 a. In some embodiments,tables such as those described herein may be used to store and managesystem metadata in addition to user data. In various embodiments, thetables maintained by the systems described herein may not have fixedschemas. As such, items may not include placeholders (i.e. emptyelements) for attributes that are not included therein, and attributes(and their values) may be added to one or more items without having toadd them to all other items.

In some embodiments, a table maintained by the data storage service onbehalf of a client/user may have a primary key that identifies itsitems. The primary key may be defined over one attribute (and may besingle valued, as described above) or over several attributes (i.e. itmay be a composite primary key, as described above), in variousembodiments. The key attributes may be immutable, may have a fixed type,and may be mandatory for every item, as they uniquely identify an itemwithin a table. In some embodiments, the primary key is the only part ofthe table that is indexed, and the index type may be specified when atable is created. For example, when a table of items is created, anattribute may be designated as the primary key attributes for the table(or two attributes may be designated for a composite primary key). Allitems in the table must include the attribute(s) designated for theprimary key and the data storage service (and/or underlying system) mayensure that the value (or combination of values) for those attributenames is unique for each item in the table. For example, if an attemptis made to add a new item that has the same primary key value as anexisting item, the new item may replace the existing item in the table.

As noted above, the data storage service (and/or the underlying system)may create an index based on the primary key. The type of index may bedependent on the whether the table uses a simple primary key or acomposite primary key. For example, the data storage service may indexthe primary key either as a hash index or a hash-and-range index, asfollows:

-   -   Hash—A hash may be a string or a number. Simple primary keys may        have one index value: a hash index (which may be a string or a        number).    -   Range—A range may be a string or a number. A range may allow        table items to be sorted so that data queries can refine results        based on the range. Composite primary keys may contain two        values for the index: a hash index (sometimes referred to herein        as the hash key value) and a range index (sometimes referred to        herein as the range key value).

A simple primary key may be sufficient for data collection andinfrequent scanning of table data (e.g., using the scan API describedbelow). A composite primary key may allow table data to be organizedmore precisely, and may allow the use of a Query API (such as thatdescribed below) for more efficient data retrieval. The followingaddress table (Table 1) illustrates the use of a single attribute as theprimary key to uniquely identify each item in the table.

TABLE 1 uses a simple primary key (string) Primary Key Other AttributesUserID = Jennifer street = 100 Pine, city = Seattle, state = WA UserID =Bob street = 24 Freemont Ave, zip = 95112 UserID = Harold street = 20104N. 4^(th) St., suite = 35, city = Columbus, state = OH

In this example, the primary key, an attribute called UserID, isrequired in every item and its type (“string”) is fixed for every item.However, each item may also include any combination of additionalattributes. The data storage system may in some embodiments beconfigured to ensure that the value of UserID is unique for each item inthe table. As noted above, in some embodiments, attribute values cannotbe null or empty. In such embodiments, an attribute does not exist inthe table until/unless it has a value associated with it.

The data storage service described herein (and/or the underlying system)may provide an application programming interface (API) for requestingvarious operations targeting tables, items, and/or attributes maintainedon behalf of storage service clients. In some embodiments, the service(and/or the underlying system) may provide both control plane APIs anddata plane APIs. The control plane APIs provided by the data storageservice (and/or the underlying system) may be used to manipulatetable-level entities, such as tables and indexes. These APIs may becalled relatively infrequently (when compared to data plane APIs). Insome embodiments, the control plane APIs provided by the service may beused to create tables, delete tables, and/or describe tables. In someembodiments, control plane APIs that perform updates to table-levelentries may invoke asynchronous workflows to perform a requestedoperation. Methods that request “description” information (e.g., via adescribeTables API) may simply return the current known state of thetables maintained by the service on behalf of a client/user. The dataplane APIs provided by the data storage service (and/or the underlyingsystem) may be used to perform item-level operations, such as storing,deleting, retrieving, and/or updating items and/or their attributes, orperforming index-based search-type operations across multiple items in atable, such as queries and scans.

The APIs provided by the service described herein may support requestand response parameters encoded in one or more industry-standard orproprietary data exchange formats, in different embodiments. Forexample, in various embodiments, requests and responses may adhere to ahuman-readable (e.g., text-based) data interchange standard, (e.g.,JavaScript Object Notation, or JSON), or may be represented using abinary encoding (which, in some cases, may be more compact than atext-based representation). In various embodiments, the system maysupply default values (e.g., system-wide, user-specific, oraccount-specific default values) for one or more of the input parametersof the APIs described herein.

As noted above, the control plane APIs supported by the service mayinclude APIs that perform updates on tables (e.g., a CreateTable APIand/or a DeleteTable API). In various embodiments, these APIs may invokeasynchronous workflows to perform the requested operation. In addition,the service may support methods that return the current known state(e.g., a DescribeTables API). In some embodiments, a common use modelmay be for a client to request an action (e.g., using a CreateTableAPI), and then to poll on its completion via the correspondingdescription API (e.g., DescribeTables).

In various embodiments, a CreateTable API may be used to create a tablehaving a specified primary index (i.e. a primary key). In someembodiments, in response to receiving a request to create a table onbehalf of a storage service client via this API, the service may trigger(and/or the underlying system implementing the service may invoke) anasynchronous CreateTable workflow that returns immediately (i.e. withoutwaiting for the workflow to be completed). In such embodiments, thesuccess of the workflow may be subsequently determined by checking thestatus of the table via a DescribeTables API. For example, each tablebeing managed by the service on behalf of a client/user may be in one ofthe following table states, and an indication of the state of each tablemay be returned in a response to a DescribeTables request:

-   -   Creating—in which the table is being created    -   Active—in which the table exists    -   Deleting—in which the table is being deleted

In some embodiments, in response to receiving a request to create atable on behalf of a storage service client/user (e.g., using aCreateTable API), the data storage service (and/or the underlyingsystem) may in some embodiments generate metadata to be associated withthe table and invoke an asynchronous CreateTable workflow to create thetable. In some embodiments, there may be multiple tables storing and/ormaintaining metadata associated with table creation, and one or more ofthese tables may be updated with when a new table is created. Forexample, the system may maintain metadata in any or all of the followingtypes of tables, or may maintain metadata in other types and numbers oftables, in various embodiments:

-   -   Tables Table: This table may maintain a list of every table in        the system, along with the current state of the table (e.g.,        Creating, Active, Deleting, etc). The primary key for this table        may in some embodiments include a Subscriberld attribute (which        may be used to identify the user on whose behalf the table will        be maintained) and a TableName attribute (which may specify the        name of the table that will be created). When an entry is        created for the new table, the table status may be set to        “Creation Pending”, which may indicate that the table has been        accepted for creation, but that a workflow has not yet been        invoked to create the table.    -   Subscribers Table: This table may maintain a count of the total        number of tables being maintained on behalf of a single client        (i.e. user/subscriber or client application), and may also        indicate how many of them are in each of the states Active,        Creating, and/or Deleting. The primary key for this table may in        some embodiments include a Subscriberld attribute, as described        above. In some embodiments, this table may be treated as a        secondary index to the Tables table. The count of the total        number of tables and/or the count of the number of tables in the        Creating state may be incremented in response to the invocation        of a CreateTable workflow.    -   Partitions Table: This table may maintain a list of all        partitions for a particular table, and may indicate their        locations. The primary key for this table may in some        embodiments include a TableId attribute and a PartitionId        attribute.    -   Nodes Table: This table may maintain a list of nodes, and may        indicate the partitions that are hosted on each of them. The        primary key for this table may in some embodiments include a        NodeId attribute. In some embodiments, this table may be treated        as a secondary index to the Partitions table.

As previously noted, a system that is configured to implement the datastorage service described herein may rely on one or more workflows thatare executed using an external workflow service. FIG. 4 illustrates oneembodiment of such a workflow for creating a table. As illustrated at410, the method may include invoking the CreateTable Workflow (e.g., inresponse to a request to create a table, and subsequent to generatingmetadata for the new table). As illustrated in this example, in someembodiments, the table name, table identifier, and/or partitionidentifiers may all be passed to the CreateTable workflow as inputs tothat process. Note that this (and/or any other service requestsdescribed herein) may include an input parameter identifying aparticular subscriber, such as an accountlD parameter. In suchembodiments, the value of this input parameter may be passed to anyworkflows invoked in response to receiving the service request (e.g.,the CreateTable workflow).

In some embodiments, a CreateTable workflow may allocate one or morepartitions for a new table, create two or more replicas each for thepartitions, and update the appropriate metadata in response to creatingthe table. One embodiment of such a workflow is illustrated by the flowdiagram in FIG. 4. The workflow may be intended to be self-healing, insome embodiments. In such embodiments, if the process fails beforecompletion, the whole workflow may be rerun one or more times until itsucceeds. For example, each of the operations illustrated in FIG. 4 maybe retried again and again in response to a failure. Note that in thisexample, it is assumed that the workflow is invoked only afterdetermining that no active table exists that has the specified tablename.

As illustrated in this example, the workflow may include updating thestatus of the table to “Creating” to reflect the fact that a workflow iscurrently working to create the table, as in 420. In some embodiments,the table status may be atomically updated to “Creating”. In suchembodiments, if multiple workflows attempt to perform this same tablecreation operation, only one will succeed, thus allowing the system toavoid a race condition, in this case. The workflow may also includedetermining whether any old partitions exist that include the table namespecified for the new table, as in 430. For example, if a creationoperation specifying this table name has been attempted (and failed) inthe past, there may be remnant partitions remaining in the system thatshould be deleted before proceeding with the rest of the CreateTableworkflow. In some embodiments, the workflow may include queryingmetadata (e.g., the Tables table) for any partitions associated withthis table name. For example, there may be remnants of a previous failedattempt to create a table with this table name in the system, includingmetadata for the table in one or more metadata tables. For eachpartition found, there may be multiple replicas, and each of thesereplicas may be physically deleted from the storage nodes on which theyreside, as in 435.

If no partitions associated with the specified table name are found(e.g., if this table creation operation has not been previouslyattempted and failed), shown as the negative exit from 430, or once suchremnants have been deleted, the workflow may create one or morepartitions for the new table, as in 440. As previously described, insome embodiments, the number of partitions created may be based on userinput, historical data, and/or system-wide, client-specific, orapplication-specific defaults. As illustrated in FIG. 4, creatingpartitions for the new table may include selecting nodes on which tostore multiple replicas of each of the partitions, creating the multiplereplicas, and updating the partition metadata (e.g., updating thePartitions table to include the newly created replicas and to indicatetheir locations). In some embodiments, selecting the nodes on which tostore the replicas may include querying metadata to discover healthynodes on which replicas can be stored, and allocating replicas tovarious ones of the healthy nodes using any of a variety of suitableallocation algorithms. In some embodiments, the system may support twoor more flexible and/or pluggable allocation algorithms, including, butnot limited to, selecting the nodes that have the most available storagespace, selecting the nodes experiencing the lightest workload (e.g., thenodes receiving the fewest service requests), or selecting nodes atrandom (which may minimize a herding effect in which all new partitionsgo to the most lightly loaded nodes).

As illustrated in FIG. 4, the CreateTable workflow may include updatingnode related metadata for the newly created table (e.g., in the Nodestable), as in 450. For example, the workflow may include reading all ofthe node locations of the newly created replicas from the Partitionstable (which was updated in 440), and adding each of the newly createdreplicas to the appropriate entries of the Nodes table. Once the table'spartitions (and their replicas) have been created, and the appropriatemetadata has been updated to reflect the creation of the new table, theworkflow may include updating the status of the newly created table to“Active”, as in 460. In some embodiments, updating the status of thenewly created table to “Active” may include decrementing a count of thenumber of tables that are in the Creating state in the Subscribers tabledescribed above.

As noted above, in some embodiments, if any of the operationsillustrated in FIG. 4 fail, they may be retried up to a pre-determinedmaximum number of attempts. For example, in one embodiment, anyCreateTable workflow step that is unsuccessful may be retried up to tentimes, and may employ an exponential back-off between attempts. In someembodiments, if the workflow step is not successfully completed afterthe maximum number of attempts, the state of the table being created maybe reset to Creation Pending to indicate that no workflow is currentlyworking on creating the table. In such cases, the system may or may notperform cleanup of any residual replicas created during the unsuccessfulattempts. For example, in some embodiments, this cleanup may be left fora subsequent CreateTable workflow. In some embodiments, a sweeperworkflow may run periodically (e.g., once every 30 minutes), and mayscan the Tables table to determine if there are any tables currently instate Creation Pending. If so, and if the state of this table has notbeen updated since the last time the Tables table was scanned by thesweeper workflow, the sweeper workflow may assume that the creation ofthis table failed, and may invoke a new CreateTable workflow in anattempt to create the table.

In various embodiments, a DeleteTable API may be used to delete a tableand all of its indexes. In some embodiments, if a table that is thetarget of a DeleteTable API is in a Creating state when the request todelete to that table is received on behalf of a storage service client,the service may return an indication of an error (e.g., a 400“ResourceInUse” error indication). If the table is in an Active statewhen the request is received, the service may trigger (and/or theunderlying system implementing the service may invoke) an asynchronousDeleteTable workflow that returns immediately (i.e. without waiting forthe workflow to be completed). In such embodiments, the success of theworkflow may be subsequently determined by checking the status of thetable via a DescribeTables API. In various embodiments, a DescribeTablesAPI may be used to enumerate (e.g., list) information about tablesbelonging to a given storage service client. For example, in response toreceiving a request on behalf of a user to describe tables belonging tothat user, the data storage system may return primary key informationand/or the status of any tables specified in the request or (if none arespecified) all tables that belong to that user. If the indication of thestate of the table that is returned in a response to a DescribeTablesrequest is “Deleting” then the delete operation may be in progress. Insome embodiments, no error indication would be returned in this case.Once the delete process is complete, the response to a DescribeTablesrequest may no longer include an entry for the deleted table.

As noted above, the data storage service (and/or underlying system)described herein may provide various data plane APIs for performingitem-level operations, such as a PutItem API, a GetItem (or GetItems)API, a DeleteItem API, and/or an UpdateItem API, as well as one or moreindex-based seek/traversal operations across multiple items in a table,such as a Query API and/or a Scan API.

In some embodiments, a PutItem API may be used to insert a new (single)item in a table. In some embodiments, this API may be used to perform aconditional put operation. For example, it may be used to insert an itemin a table if it does not already exist in that table (according to thespecified value of the primary key), or to replace an existing singleitem in a table if it has certain attribute values (e.g., a specifiedprimary key). More specifically, in some embodiments this API may beused to completely replace all of the attributes of an existing item(except the primary key) with new attributes to create a “new” item. Insuch embodiments, the data storage system may guarantee that thisreplacement operation is performed atomically. In other words, thesystem may perform the replacement operation in a way that guaranteesthat the item is observable only with all of its new attributes or withall of its previous attributes, and is not observable in an interimstate (e.g., with a mix of previous and new attributes). In someembodiments, the PutItem API may be an idempotent API if a conditionalput operation is not specified. In other words, a request made using anon-conditional form of the PutItem API may insert a specified new itemin a table exactly once, even if it is called multiple times with thesame input parameter values.

In various embodiments, a DeleteItem API may be used to delete a singleitem in a table, where the item is identified by its primary key. Insome embodiments, this API may be used to perform a conditional deleteoperation. For example, it may be used to delete an item if it exists,or if it has certain attribute values (e.g., particular attribute valuesother than the specified primary key). In some embodiments, theDeleteItem API may be an idempotent API if a conditional put operationis not specified. In other words, a request made using a non-conditionalform of the DeleteItem API may cause the system to delete a specifiednew item in a table exactly once, even if it is called multiple timeswith the same input parameter values. In these and other embodiments,attempting to delete a non-existent item may not result in an errorcondition, and may not cause an error indication to be returned.

In various embodiments, a GetItem or GetItems API may be used toretrieve one or more items (i.e. to return one or more attributes ofthose item), given their primary keys. In some embodiments, the numberof items that can be retrieved in response to a single GetItems requestmay be limited and/or the items retrieved must all be stored in the sametable. For example, in one embodiment, attributes for a maximum of eightitems may be returned in response to a single GetItems request. In someembodiments, multiple items may be retrieved from a table in parallel,which may minimize latency. The data storage service (and/or theunderlying system) may support projection and/or consistent reads(without a latency penalty), in various embodiments. In someembodiments, the system may support an eventual consistency model bydefault, which may result in higher throughput for servicing requests.In some embodiments in which multiple items are requested in a singleGetItems request, items that do not exist in the targeted table will notbe returned. In this case, there may or may not be any error messagesreturned to indicate that one or more of the requested items were notreturned.

In various embodiments, an UpdateItem API may be provided by the datastorage service (and/or the underlying system). This API may be used toinsert an item if it does not already exist, or to manipulate anexisting item at the attribute level (e.g., to modify the values of oneor more of its attributes). For example, updating an item may includeinserting, replacing, and/or deleting various attributes of an existingitem. In some embodiments, updating an item may include atomicallyincrementing or decrementing the value of an attribute having a numbertype. While the PutItem API described above may be used to replace allof the attribute values of an existing item, the UpdateItem APIdescribed herein may provide a more granular replacement operation. Inother words, this API may be used to modify a subset of the attributevalues of an existing item, and/or to modify the set of attributes thatare defined for an existing item.

In various embodiments, an UpdateItem API provided by the data storageservice (and/or the underlying system) may perform a conditional update.In such embodiments, this API may be used to conditionally insert anitem (e.g., to create an item if it does not already exist), or toconditionally replace (i.e. update) an item (e.g., only if itsattributes match any specified expected values). Updating an item mayinclude inserting, updating, and/or deleting various attributes of anexisting item. In some embodiments, the data storage system mayoptionally return the old attribute values for an item that isreplaced/updated using this API.

As previously noted, in embodiments in which the primary key is a simplekey, the item in a table being maintained on behalf of a storage serviceclient may partitioned using a hash of the primary key value of each ofthe items, while in embodiments in which the primary key is a compositekey, the data may be partitioned first by a hash of the hash keycomponent, and then by the range key component. FIG. 5 illustrates oneembodiment of a method for partitioning table data using simple and/orcomposite keys, according to one embodiment. As illustrated at 510, inthis example, the method may include a data storage service (or acomponent of the underlying system that implements a data store, such asa storage node instance or administrative component) initiating thepartitioning of a table maintained in a non-relational data store onbehalf of a storage service client.

If multiple items in the table share a hash key attribute value, shownas the positive exit from 520, the method may include the data storedividing the items in the table that have a given hash key attributevalue into two or more partitions (e.g., database partitions) dependentfirst on a hash of their range key attribute values, and then on theirrange key attribute values, as in 540. In other words, if the primarykey for the table is a composite key that includes hash key componentwhose values may be used to identify a group of items and a range keycomponent whose values may be used to order items having the same hashkey attribute values and uniquely identify each of those items, both thehash key attribute value and the range key attribute value may be usedto partition the items in the table. For example, for a group of itemsthat have the same hash key attribute value, the first n items in thegroup (when ordered by their respective range key attribute values) maybe assigned to one partition, the next m items in the group may beassigned to a second partition, and so on. Note that in someembodiments, each partition may include a portion of the items sharingone hash key attribute value and may also include other items havingother hash key attribute values.

If none of the items in the table share a hash key attribute value,shown as the negative exit from 520, the method may include the datastore dividing the items in the table into two or more partitionsdependent on a hash of their respective hash key attribute values, as in530. For example, if the primary key for the table is a simple key thatincludes hash key component whose values may be used to uniquelyidentify each of the items in the table, the items in the table may bepartitioned (i.e. assigned to one of a plurality of partitions)dependent a hash of the hash key attribute value, but not dependent onany other item attribute values. In some embodiments, if the primary keyis a composite key, but none of the items in the table share a hash keyattribute value (i.e. if each item has a unique hash key attributevalue), the data store may partition the items as if the primary keywere a simple key (i.e. it may partition the items in the table usingthe hash key attribute value alone).

Once the data store has assigned all of the items to a partition, thedata store may store each of the partitions on a respective storage node(e.g., a respective computing node or storage device), as in 550. Insome embodiments, each partition of a single table may be stored on adifferent storage node, while in other embodiments two or more of thepartitions may be maintained on the same storage node. In variousembodiments, each of the resulting partitions may be replicated one ormore times in the data storage system, as in 560. Note that in someembodiments, the number of partitions into which the items of a giventable are partitioned may be pre-determined (e.g., it may be based onuser input/preferences, or historical data for a client, account, ortable type), while in other embodiments, the number of partitions intowhich the items of a given table are partitioned may be determined asthe partitioning operation progresses, e.g., based on the number ofitems in each range of hash results and/or the number of items in eachrange of range key attribute values. Note also that because thepartitioning is based on a hash result, the order in which groups ofitems may be assigned and distributed among the available partitions maybe somewhat randomized. In some cases, e.g., if some items are accessedmuch more frequently than others or some groups of items include ahigher number of items than others, an initial partitioning may resultin hot spots. In such cases, a repartitioning operation may be performedin order to more evenly distribute the items among the availablepartitions (e.g., with respect to data volume and/or service requesttraffic). Note also that in some embodiments, the items in a table maybe partitioned using a single hash key component and two or more rangekey components.

Table 2 below illustrates an example of the partitioning of items intable using a method similar to that illustrated in FIG. 5. In thisexample, the hash key attribute is a “User name” attribute, and therange key attribute is a “Message ID” attribute. The table storesmultiple messages associated with each of three user names (Bob, Sue,and Phil). As illustrated in Table 2, some partitions of a given tablemay include only items having the same hash key attribute value. In thisexample, a partition identified by a Partition ID value of A stores onlymessages having the hash key attribute value “Bob”. Note that thispartition does not store all of Bob's messages, only messages havingMessage ID values (i.e. range key attribute values) 1-199. Another groupof Bob's messages (those with range key attribute values 200-299) arestored in a partition identified by a Partition ID value of B. Thispartition also stores messages having a hash key attribute value of“Sue”, specifically, those messages having range key values of 1-50. Yetanother group of Bob's messages (those with range key attribute values300-399) are stored in a partition identified by a Partition ID value ofC. This partition also stores messages having a hash key attribute valueof “Phil”, specifically, those messages having range key values of1-100.

TABLE 2 User name Message ID Partition ID Bob  1 A Bob  2 A . . . Bob199 A Bob 200 B . . . Bob 299 B Bob 300 C . . . Bob 399 C Sue  1 B Sue 2 B . . . Sue  50 B Phil  1 C Phil  2 C . . . Phil 100 C

In the example above, a request to retrieve all of Bob's messages mayretrieve messages 1-199 from partition A (which may be maintained on aparticular storage node), messages 200-299 from partition B (which maybe maintained on a different storage node), and messages 300-399 frompartition C (which may be maintained on yet another storage node). Asdescribed in more detail herein, in some embodiments, a request toretrieve all of these messages may be terminated early (e.g., ifresponse limit is reached), and the remaining messages may be retrievedin response to a subsequent request.

In some embodiments, the data storage service (and/or underlying system)described herein may provide two different APIs for searching the datamaintain in tables on behalf of storage service clients: a Scan API anda Query API. In some embodiments, the Scan API may be used to request anoperation that scans an entire table. A Scan request may specify one ormore filters to be applied to the results of the scan operation, e.g.,to refine the values returned to the requestor following the completescan. In some embodiments, the service (and/or underlying system) mayimpose a limit on the scan results, and the limit may be applied beforethe results are filtered. For example, in some embodiments, the systemmay use pagination (e.g., dividing a scan or query process into distinctpieces having a pre-determined maximum size in terms of the number ofitems evaluated or returned, or in terms of the amount of data scannedor returned) in order to respond to scans and/or queries quickly. Forexample, in order to scan a table that is larger than the pre-determinedmaximum size (e.g., 1 MB) or for which the resulting data set is largerthan a pre-determined maximum size (e.g., 1 MB), multiple scan or queryoperations may need to be performed to scan the entire table, in 1 MBincrements. It may be possible for a scan operation to return noresults, if no table data meets the specified filter criteria.

In some embodiments, the Query API may support comparison operations tolimit the search process to the data that matches the supplied queryconditions (e.g., conditions on the attributes of the items). Forexample, a Query request may be used to find all the data in a tablethat matches the parameters specified in the request, up to apre-defined limit (if such a limit is imposed by the system). In someembodiments, a Query request may always returns results, but the systemmay return empty values if the query conditions (i.e. the attributefilter criteria) does not match any of the results. In variousembodiments, a Query API may be used to query a table that is maintainedon behalf of a storage service client (e.g., a user, customer,subscriber, or client application) for information stored in that table.In some embodiments, the query may be performed based on a primary index(according to a specified hash key and, in some cases, a single rangekey value that satisfies a specified range key predicate). In otherembodiments a primary key may include a single hash key component andtwo or more range key components.

A more detailed example of a method for performing a query, as specifiedby the API described herein, is illustrated by the flow diagram in FIG.6, according to one embodiment. As illustrated at 610, in this example,the method may include receiving a service request to perform a querythat is directed to one or more items in a table in a non-relationaldatabase (e.g., a table maintained on behalf of a data storage serviceclient). As in previous examples, the request may include a table name(which may identify the table that is the target of the query), and aprimary key value. In this example, the specified primary key value is acomposite key value (i.e. the primary key for the identified table is acomposite primary key dependent on a hash key value and a range keyvalue), and the query may target multiple items that match the hash keyvalue and range key condition specified in the request, as describedherein. As illustrated at 620, the method may include parsing therequest to determine the hash and range values specified in the request.

The method may include directing the query to a partition that comprisesan initial target of the query, dependent on the specified hash andrange values, and retrieving information about one or more targets ofthe query (e.g., attribute values of the items targeted by the query)from that partition, as in 630. For example, in some embodiments, theitems matching a particular hash key value may be ordered in the tableby their range key values. In such embodiments, the combination of thespecified hash key value and the first range key value that matches thespecified range key condition may uniquely identify the first item inthe table that matches the query conditions. In such embodiments, aquery may first be directed to the partition that contains the itemidentified by this combination. In some cases, one or more additionalitems matching the specified hash key value and the specified range keycondition may be present on the first partition to which the query isdirected, and all of these targets (i.e. the items themselves and/or aspecified subset of their attribute values) may be returned in responseto the query.

In some cases, some of the items matching both the specified hash keyvalue and the specified range key condition may be stored on one or morepartitions of the table other than the first partition to which thequery was directed. If so, shown as the negative exit from 640, thequery may be directed to the one or more other partitions, and theseadditional query targets may be retrieved, as in 650. For example, thenumber of items matching both the specified hash key value and thespecified range key condition may be larger than the number of itemsstored in each partition of the table. In another example, because ofthe order in which items are sorted and stored in the table and/orassigned to various partitions (e.g., in embodiments in which items aresorted in a particular order and assigned to a particular partitionaccording their range key values), the targeted items may cross apartition boundary. In these and other cases, the method may includereturning a response that includes one or more attribute values of oneor more items matching both the hash key value and the range keycondition, as in 670, where some of the one or more items matching boththe hash key value and the range key condition may be retrieved fromdifferent partitions (and, in some cases, different physical computingnodes or storage devices).

As illustrated in FIG. 6, however, if all of the items matching both thespecified hash key value and the specified range key condition arestored on the first partition to which the query was directed, shown asthe positive exit from 640, the method may include returning a responsethat includes one or more attribute values of one or more items matchingboth the hash key value and the range key condition, as in 660, whereall of the one or more items matching both the hash key value and therange key condition are retrieved from the initially targeted partition(and, thus, a single physical computing node or storage device).

In various embodiments, a Scan API may be used to retrieve one or moreitems and attributes stored in a table on behalf of a storage serviceclient by performing a full scan across the table. The items returnedmay be limited by specifying a filter. In some embodiments, the Scan APImay support richer semantics than the Query API described above. Forexample, it may support comparison operators such as “CONTAINS”, “ISNULL”, “IN”, etc.

Note that in some embodiments, the following error indications may bereturned by any of the APIs supported by the service, while others maybe returned by specific ones of these APIs.

-   -   InvalidParameterValue    -   MissingParameterValue    -   InternalFailure    -   ServiceUnavailable

In some embodiments, any or all of the metadata described herein asbeing used in maintaining and managing tables on behalf of a datastorage service client (including any of the metadata tables describedherein) may be stored in the same scalable data store (e.g., the samenon-relational database) as that in which the client/user tables arestored. In such embodiments, the system may include or employ one ormore bootstrapping mechanisms to assist with initialization of the datastorage service (and/or the underlying system that implements a datastorage service), some of which are described herein. FIG. 7 illustratesa portion of a data model for such a system, according to oneembodiment. In this example, various computing nodes (represented in thedata model simply as “nodes 710”) may store user data (e.g., in tablesmaintained on behalf of a user) and/or system data, including metadataused by the data storage service, such as that described above.Therefore, each node 710 of the data model may include an indicator ofthe type of the node, shown as node-type 715. For example, in oneembodiment, each node may be designated as a “storage node”, a “requestrouter”, an “auto-admin” node, or a “staging” node. In some embodiments,a “storage node” may store user data in one or more tables maintained bythe data storage service, but metadata (e.g., data stored in one or moreof a Tables Table, a Subscribers Table, a Partitions Table, or a NodesTable) may be hosted on other types of nodes (e.g., “auto admin” nodesand/or “staging” nodes). In other embodiments, such metadata may bestored on one or more “storage nodes”, some of which may also store userdata. As illustrated in FIG. 7, each node 710 may also include anidentifier of the node (shown as node-id 720), and one or more otherelements (shown as 730).

As illustrated in FIG. 7, information about each replica may berepresented in the data model as a replica 740. Each replica 740 in thedata model may include an identifier of the node on which the replica ishosted (shown again as node-id 720), and one or more partitionidentifiers (shown as partition-id 735) indicating the partitionsincluded in those replicas. In this example, each partition may berepresented in the data model as a partition 750 and may include itspartition-id 755. As illustrated in FIG. 7 by various one-to-manymappings, each node may host multiple replicas, and each partition maybe included in multiple replicas.

In some embodiments, the systems described herein may support seamlessscaling of user tables in a “fully shared nothing” type architecture.For example, in some embodiments, each partition may be implemented as acompletely independent parallel computation unit. In such embodiments,the system may not provide distributed coordination across partitions orsupport batch “put” operations and/or multi-statement transactions. Insome embodiments, as long as the workload distribution is well spreadacross partitions, an increase in the number of partitions may result ina larger usable table size and/or increased throughput capacity forservice requests. As described herein, in some embodiments, liverepartitioning (whether programmatic/automatic or explicitly initiated)may be employed to adapt to workload changes. In other words, in someembodiments, repartitioning (including partition moving, partitionsplitting, and other repartitioning operations) may be performed whileservice requests directed to the affected partitions continue to bereceived and processed (i.e. without taking the source partitionoff-line).

In different embodiments, the data storage service (and/or underlyingsystem) may support a variety of service offerings and/or throughputmodels. For example, in some embodiments, the service may support acommitted throughput offering and/or a best effort offering. In someembodiments, a storage service client (e.g., a client application, user,or subscriber having access to the service) may specify a preferencebetween multiple throughput options that are offered by the service,according to a variety of business models, subscription types, and/orpayment models. For example, the client/user may indicate a preferredthroughput model for a particular table through a parameter of a requestto create the table, in some embodiments. In other embodiments, aclient/user may specify a default throughput model for all tablescreated and maintained on their behalf by the data storage service. Bysupporting both a committed throughput model and a best effortthroughput model (for which no throughput guarantees are made), thesystem may allow clients/users to make a trade-off between performanceand cost, according to their needs and/or budgets.

A data storage service (and underlying system) that provides a committedthroughput offering may be configured to pre-allocate capacity and/orresources for the creation, growth, and management of a table maintainedon behalf of a client/user in response to traffic directed to the table,and not to overbook the resources and/or capacity of the storage node(s)on which that table is maintained. In some embodiments, tablesmaintained by the service (and underlying system) under a committedthroughput model may be maintained in faster (and often more expensive)storage resources, such as high performance media (e.g., flash memory orSolid State Drive, or SSD, media), in order to provide extremely lowlatencies when servicing requests from the client/user. For example, thesystem may provide (and dedicate) a high ratio of fast/local memory tomain (e.g., disk) memory for the maintenance of those tables (andvarious partitions thereof). While the storage resources allocated to agiven table under a committed throughput model may in some cases beunderutilized (at least some of the time), the client/user may value thepredictable performance afforded by the committed throughput model morethan the additional (and in some cases wasted) costs of dedicating moreresources than may always be necessary for that table.

In various embodiments, there may be situations in which a partition (ora replica thereof) may need to be copied, e.g., from one machine toanother. For example, if there are three replicas of a particularpartition, each hosted on a different physical or logical machine, andone of the machines fails, the replica hosted on that machine may needto be replaced by a new copy (replica) of the partition on anothermachine. In another example, if a particular machine that hosts variousreplicas of multiple partitions of one or more tables experiences heavytraffic, one of the heavily accessed partition replicas may be moved(e.g., using a copy operation followed by an operation to redirecttraffic) to a machine that is experiencing less traffic in an attempt tomore evenly distribute the system workload and improve performance. Insome embodiments, the data storage service (and/or underlying system)described herein may perform replica moves and/or replica copying usinga physical copying mechanism (e.g., a physical file system mechanism)that copies an entire partition replica from one machine to another,rather than copying a snapshot of the partition data row by row (as in atraditional logical database partition copying operation). As describedin more detail herein, in some embodiments, all write operations may belogged before being applied to a particular partition (and/or variousreplicas thereof), and they may be applied to the partition (i.e. to thereplicas thereof) periodically (e.g., in batches). In such embodiments,while a partition replica is being copied, write operations targetingthe partition may be logged. During the copy operation, these loggedwrite operations may be applied to the partition at periodic intervals(e.g., at a series of checkpoints). Once the entire partition has beencopied to the destination machine, any remaining logged write operations(e.g., any write operations performed since the last checkpoint, or anywrite operations that target portions of the partition that were copiedto the destination prior to those write operations being logged) may beperformed on the destination partition replica by a final catch-upprocess. In some embodiments, the catch-up process may examine thesequentially ordered write operations in the log to determine whichwrite operations have already been applied to the destination partitionreplica and which, if any, should be applied to the destinationpartition replica once the physical copying of the partition data iscomplete. In such embodiments, unlike with traditional partition copyingor moving mechanisms, the data in the destination partition replica maybe consistent following the completion of the operation to move/copy thepartition replica.

One embodiment of a method for moving (or copying) a replica of apartition of a table being maintained by a data storage service onbehalf of a storage service client while the partition is “live” isillustrated by the flow diagram in FIG. 8. In this example, the methodmay include a component of the system that implements the data storageservice receiving a request to move a replica of a partition, as in 810.For example, the system may receive an explicit request to move areplica from a client/user or system administrator, or such a requestmay be automatically generated in the system in response to detecting ananomaly (as described in more detail herein). As illustrated at 820, inresponse to receiving the request to move the partition, the system maybe configured to create a new replica (which may be referred to as adestination replica), while the partition is live (i.e. while one ormore replicas of the partition continue to accept and service requestsdirected to the partition). In some embodiments, creating a destinationreplica may include selecting a computing node or storage device onwhich to create the destination replica, allocating memory on thecomputing node or storage device for the destination replica, creatingor updating metadata associated with the partition and/or thedestination replica, and/or performing other functions appropriate forcreating the destination replica.

As illustrated in this example, the method may include the systemcopying table data from the replica being moved (or from another sourcereplica storing the same table data as the replica being moved) to thedestination replica using a file copy mechanism or another physical copymechanism while one or more replicas of the partition are live, as in830. In other words, the replica may be copied to the new destinationreplica using an operation that copies the physical locations of thereplica data, rather than using a logical copying operation (e.g., onethat reads and copies table data on a row-by-row basis). As illustratedat 840, after performing the physical copying operation, the method mayinclude the system performing a catch-up operation to reconcile anychanges to the replica data that were made during the copy operation butthat are not yet reflected in the new copy. This catch-up operation isdescribed in more detail below. Once the destination replica has beencreated and populated, the method may include directing traffic awayfrom copied replica and toward the new designation replica, as in 850.For example, the system may configure the new destination replica toreceive and service requests targeting table data that was maintained onthe particular partition replica and some or all service requeststargeting the partition may be directed away from the source replica andtoward the new destination replica.

In some embodiments, the storage engine for the underlying data store ofa data storage service (e.g. a non-relational database) may storereplica data in database files, and each replica (and database file) maybe associated with a recovery log. In such embodiments, when a servicerequest to modify the replica data is received, it may be logged in therecovery log before being applied to the replica. In the case of a nodefailure or system crash, the changes logged in the recovery log may bereapplied to a previous snapshot or checkpoint of the replica data torecover the contents of the replica. As noted above, in someembodiments, the data storage service (and its underlying system) maysupport a replica move operation and/or a replica copying operation thatemploys a physical copy mechanism. In some such embodiments, thephysical copy mechanism may employ such a log, which may ensure that thereplica data that is moved to a new destination is consistent. FIG. 9illustrates one embodiment of a method for copying a replica using aphysical copy mechanism, as described above. In this example, the methodbegins copying replica data from its current physical storage locationsto corresponding physical destination locations, as in 910. In someembodiments, the physical copy operation may include copying pages fromone physical storage device (e.g., disk storage) to a destinationstorage device over a network.

As illustrated at 920, during the physical copying operation, writeoperations targeting the partition whose replica is being copied may belogged before being applied to the replica being copied, as describedabove. In various embodiments, each logged write operation (or group ofwrite operations) may be assigned a log sequence number. In someembodiments, the logged changes may be applied to the replica beingcopied (and/or to other replicas that store the same table data) atperiodic checkpoint intervals. In the example illustrated in FIG. 9,when a pre-determined checkpoint interval passes, shown as the positiveexit from 930, all of the modifications (e.g., write operations) loggedsince the last checkpoint may be applied to the replica being copied(e.g., the source replica) and/or to other replicas that store the sametable data. Because these updates are applied while the source replicais being copied, some of these modifications will be reflected in thedestination replica as a result of the copying operation (e.g.,modifications that were applied to a given portion of the replica databefore that portion of the data was copied to the destination). Othermodifications may not be reflected in the destination replica followingthe copying operation (e.g., modifications that were applied to a givenportion of the replica data after that portion of the data was copied tothe destination).

As illustrated in FIG. 9, the method may include continuing to copyreplica data from current physical storage locations to correspondingphysical destination locations while it is not complete (shown as thenegative exit from 950, element 960, and the feedback to 920). Themethod may include continuing to log write operations (as in 920) and toapply logged write operations to the source replica, i.e., the replicabeing copied, (as in 940) each time the checkpoint interval passes(shown as the positive exit from 930). Once the physical copy operationis complete (shown as the positive exit from 950), the method mayinclude performing a catch-up operation, in which any logged writeoperations that are not already reflected in the destination replica areapplied to the destination replica, as in 970. Thereafter, if thecopying of the partition was performed as part of an operation to movethe partition replica, some or all accesses targeting the partitionwhose replica was copied may be directed away from the source replicaand directed toward the new destination replica. For example, any writeoperations targeting the partition may be logged in a recovery log forthe destination replica, and subsequently applied to the destinationreplica (e.g., at the next periodic checkpoint). In some embodiments,following the copying of the replica to a new destination (e.g., as partof a move operation), the log in which modifications to the sourcereplica were logged may be copied (or used directly) for the recoverylog for the destination replica.

In some embodiments, the replica copying process described above may beemployed in partition splitting operations. For example, a partition maybe split because it is large (e.g., because it is becoming too big tofit on one machine) and/or in order to keep the partition size smallenough to quickly rebuild the partitions hosted on a single machine(using a large number of parallel processes) in the event of a machinefailure. A partition may also be split when it becomes too “hot” (i.e.when it experiences a much greater than average amount of traffic ascompared to other partitions). For example, if the workload changessuddenly and/or dramatically for a given partition, the system may beconfigured to react quickly to the change. In some embodiments, thepartition splitting process described herein may be transparent toapplications and clients/users, which may allow the data storage serviceto be scaled automatically (i.e. without requiring client/userintervention or initiation).

Note that in some embodiments, moving (or copying) a replica of apartition in a cluster may be quicker than splitting a partition,because the system may take advantage of the file copying processdescribed above for replica copying. Splitting a partition, on the otherhand, may require logically dividing the partition data in oneunderlying data structure (e.g., one B-tree) into two such datastructures (e.g., two B-trees), which is generally less efficient thanmoving an entire replica, as described above. Therefore, in someembodiments, a partition splitting process may include creatingadditional replicas of the partition, and thereafter managing only aportion of the partition data on each replica. For example, if there arethree replicas of a given partition that is to be split, the partitionsplitting process may include creating three additional copies of theentire partition (e.g., using the partition copying process describedabove). These resulting six replicas may be split into two new replicagroups of three replicas, each of which may be configured to beresponsible for handling service requests directed to half of theoriginal partition data by invoking an operation to split theresponsibilities between the replica groups. For example, following theoperation to split the responsibilities, service requests directed todata in a designated portion of the original partition may be acceptedand serviced by replicas of a given replica group, while servicerequests targeting the remaining data of the original partition may berejected by that replica. In some embodiments, the partition data forwhich a given replica is not responsible may eventually be removed(e.g., so that the memory allocated to the replica for data it no longersupports may be subsequently used to store new items in the replica), orthe memory in which it was stored may be reclaimed by the system (e.g.,so that the memory allocated to the replica for data it no longersupports may be subsequently used by another partition). Removal ofunsupported data or reclamation of memory may be performed by backgroundtasks without affecting the performance of the data storage system, andmay be transparent to clients/users.

In some embodiments, each partition may be identified by a partition ID,which may be a unique number (e.g., a GUID) assigned at the time thepartition is created. A partition may also have a version number that isincremented each time the partition goes through a reconfiguration(e.g., in response to adding or removing replicas, but not necessarilyin response to a master failover). When a partition is split, two ormore new partitions may be created, each of which may have a respectivenew partition ID, and the original partition ID may no longer be used.In some embodiments, a partition may be split by the system using asplit tool or process in response to changing conditions. For example, ascheduled task of an auto admin instance may monitor partition sizes and“heat” (e.g., traffic directed to each partition), and may applypolicies that determine when to use the splitting tool/process toperform a split. In some embodiments, the splitting tool and auto admininstance may avoid attempting two splits at the same time by employing alock manager.

In some embodiments, the monitoring component may provide a list ofpartitions that meet the split criteria to the splitting tool/process.The criteria may be based on partition size and heat, where heat may betracked by internally measured metrics (such as IOPS), externallymeasured metrics (such as latency), and/or other factors. In someembodiments, the splitting tool/process may receive a request to split apartition from the monitoring component that includes a partition ID anda version number for the partition to split, and a list of machines(e.g., machines in the same cluster or storage silo that are known to belightly loaded) for the location(s) of the new partitions/replicas.Including the version number as an input to the splitting tool/processmay ensure that the splitting tool/process does not attempt to split apartition that has already gone through one or more reconfigurationssince the last time it was evaluated against the split criteria, as thesplitting tool/process may reject the request if version number does notmatch.

One embodiment of a method for splitting a partition of a table beingmaintained by a data storage service on behalf of a storage serviceclient is illustrated by the flow diagram in FIG. 10. In this example,the method may include a component of the system that implements thedata storage service receiving a request to split a partition, as in1010. For example, the system may receive an explicit request to splitthe partition from a client/user or system administrator, or such arequest may be automatically generated in the system in response todetecting an anomaly (as described in more detail herein). As describedabove, in some embodiments, splitting a partition may involve creatingadditional replicas of the partition, dividing the resulting collectionof partition replicas into two or more new replica groups, and thendesignating each of the replica groups as managers of a respectiveportion of the original partition. Therefore, as illustrated at 1020, inresponse to receiving the request to split the partition, the system maybe configured to initiate creation of the one or more new partitionreplicas (which may be referred to as destination replicas), while oneor more of the original replicas of the source partition are live (i.e.while one or more of these replicas continue to accept and servicerequests directed to the partition). As illustrated at 1030, the methodmay include copying data from one or more source partition replicas tothe destination replicas using a physical copy mechanism (such as thatdescribed above). For example, the system may be configured to copy thetable partition data from one (or more) of the original replicas of thepartition to one or more of the destination replicas using a file copymechanism, in some embodiments. The method may also include bringing thenew replicas (once populated) up-to-date (e.g., by performing a catch-upoperation, as described above).

As illustrated in this example, the method may include propagating aspecial “write” command (i.e. a “split” command) to split the partitionby dividing the resulting collection of replicas into two or more newreplica groups and designating (and/or configuring) each replica groupas handling service requests directed to a respective portion of thesplit partition, as in 1040. In some embodiments, the system may takethe source replicas out of use briefly while the command to split thepartition replicas is propagated to the storage nodes on which theresulting collection of replicas are hosted. In other words, the systemmay not accept other service requests directed to the partition beingsplit while the split command is operating to configure the new replicagroups to receive subsequent service requests. In an example in which apartition is being split into two new partitions, the split command mayinstruct the replicas resulting from the copy operation to split in halfby designating each replica as belonging to the first half of the rangeor the second half of the range, thus forming two new replica groups. Inother embodiments, the split command may designate each replica asbelonging to one of more than two replica groups. Note that in someembodiments, the special “split” command may not require any specialdurability, while in others it may require the same durability as anyother replicated write operation, or may have a different durabilityrequirement than other replicated write operations.

As illustrated in this example, once the “split” command has beenpropagated and the new replica groups have been established, if thesystem is a single master system or a multi-master system, the methodmay include each of the new replica groups electing one or more mastersfor the replica group, as in 1050. Subsequently, the replicas in each ofthe new replica groups for the split partition (e.g., a replica groupmade up of the original replicas, a replica group made up of thedestination replicas, or a replica group made up of any other subset ofthe resulting replicas for the split partition) may handle requestsdirected to a respective portion of the original partition, as in 1060.For example, each of the replicas may reject requests for the table datathat is now out of its new smaller range, and may return an indicationthat the replica (or the node on which the replica is hosted) no longerhosts that data. As described above, in some embodiments, the system maybe configured to perform a logical reclamation of the unused portions ofthe resulting split partition replicas, as in 1070. For example, asrequests to store new items in the partition are received, these newitems may be stored in locations in the table that (following thereplica copying operation) held items stored in the original partition,but that are now being managed as part of a different partition (i.e.one of the two or more new partitions created by the split). In someembodiments, the system may employ a background process to logicallyfree up space within each of the resulting partition replicas, but thatspace may be consumed later if more items are added to the table thatare assigned to the new partition replicas according to their hash keyattribute values and/or range key attribute values. In some embodiments,a physical memory reclamation operation may be performed, which mayreturn a portion of the memory that was previously allocated to a largepartition replica prior to the split to the operating system. In suchembodiments, a de-fragmentation operation may also be performed.

As noted above, the partition moving process illustrated in FIG. 8 anddescribed above may be initiated automatically (e.g., programmatically)in response to detection of an anomaly in a system that implements adata storage service, in some embodiments. One embodiment of a methodfor moving a partition of a table being maintained by a data storageservice on behalf of a storage service client in response to detectingan anomaly is illustrated by the flow diagram in FIG. 11. As illustratedat 1110, in this example, the method may include a component of thesystem detecting a failure or fault on a physical computing node orstorage device that is hosting a replica of a partition of the table. Insome embodiments, if the partition replica hosted on the node on which afault or failure was detected was a master for its replica group, themethod may include electing a new master for the replica group, as in1120. In this example, the method may include the system initiatingcreation of a replacement partition replica while a source partitionreplica is live (i.e. while one or more of the replicas of the sourcepartition continue to accept and service requests directed to thepartition), as in 1130.

As illustrated in this example, the method may include copying a sourcepartition replica to the newly created replacement partition replicausing a physical copy mechanism (as in 1140), and performing a catch-upoperation to reconcile any changes to the partition data that are notyet reflected in the newly created replacement partition replica (as in1150). For example, the source partition replica may be copied to thereplacement partition replica using an operation that copies thephysical locations of the partition data, rather than using a logicalcopying operation (e.g., one that reads and copies table data on arow-by-row basis). In various embodiments, the partition replica on thefaulty machine may be used as the source partition replica, or one ormore other replicas for same partition (e.g., a replica in the samereplica group that is hosted on a working machine) may be used as thesource partition replica, e.g., depending type and/or severity of thedetected fault.

As noted above, the partition moving process described above andillustrated in FIGS. 8 and 9, and the partition splitting processillustrated in FIG. 10 and described above may be initiatedautomatically (e.g., programmatically) in response to detection of ananomaly in a system that implements a data storage service, in someembodiments. For example, if a hot spot develops on a particularcomputing node or storage device in the system underlying the datastorage service, the system may be configured to split a hot partitionfor which a replica is stored on that computing node or storage deviceand/or move one or more partition replicas stored on that computing nodeor storage device to another computing node or storage device.

In some embodiments, the data storage service (and/or underlying system)may be configured to detect anomalies in the system while servicingrequests from one or more storage service clients. In some embodiments,the system may be configured to automatically (e.g., programmatically)respond to the detection of various types of anomalies, such as byscaling tables, moving partitions, splitting partitions, and/or takingother actions not described herein. For example, if a failed or faultynode (e.g., a computing node or storage device) has been detected, thesystem may be configured to replace the failed or faulty node with a newnode and/or to move any or all partitions that are hosted on the failedor faulty node to the new node. As described herein, such a move may insome embodiments be performed using a physical copy operation. Aspreviously noted, if a failed or faulty node hosted a partition replicathat was a master for its replica group, the system may also beconfigured to elect a new master for the replica group subsequent tocopying the partition to the new node.

If a hot spot or increasing table/partition size is detected, the systemmay be configured to add one or more new partitions and correspondingreplicas (e.g., on computing nodes or storage devices other than the oneon which the hot spot was detected), and to move and/or split data thatwas hosted on the heavily loaded computing node or storage device in oneor more of the new partitions or replicas. Similarly, if the system hasdetected that a best effort throughput target (or another userpreference) is not being met or is in danger of not being met due toincreasing traffic or if the data volume is increasing beyond a targetedcapacity for the table, the system may be configured to throttleincoming service requests while attempting to correct the situation.Again, the system may be configured to add one or more new partitionsand corresponding replicas (e.g., on computing nodes or storage devicesother than the one on which the hot spot was detected), and to moveand/or split data that was hosted on the heavily loaded computing nodeor storage device in one or more of the new partitions or replicas.Similarly, if a live repartition is explicitly requested (e.g., by atable owner), the system may be configured to add or remove one or morepartitions and corresponding replicas accordingly, or to move and/orsplit data that was hosted on a heavily loaded computing node or storagedevice in one or more partitions or replicas.

In general, once an anomaly has been detected and the system hasresponded to and/or returned an indicator of that anomaly, the systemmay resume (or continue) servicing incoming requests. In someembodiments, the system may be configured to continue operation (e.g.,to continue servicing incoming service requests) until or unlessadditional anomalies are detected. If any additional anomalies aredetected, any or all of the operations described above for resolvingsuch anomalies may be repeated by the system in order to maintain andmanage tables on behalf of data storage service clients. Note that insome embodiments, any or all of the operations described above forresolving such anomalies may be performed pro-actively (andautomatically) by background tasks while the data storage service is inoperation, and may not necessarily be performed in response to receivingany particular service requests.

In various embodiments, the systems described herein may provide storageservices to clients, and may maintain data on behalf of clients inpartitions that are replicated on multiple storage nodes. In someembodiments, these storage systems may implement a single masterfailover protocol. In some embodiments, membership in various replicagroups may be adjusted through replicated changes, and membership andother updates in the system may be synchronized by synchronizing over aquorum of replicas in one or more data centers at failover time using areplicated quorum version. In some embodiments, a mechanism forsplitting a partition may utilize failover quorum synchronization,external master locks, and/or various methods for detecting andresolving log conflicts, including log snipping (e.g., deleting logrecords that are on invalid branches). The systems may implement afault-tolerant log shipping based replication mechanism that includessuch log conflict detection and resolution. In some embodiments, logbranching may be avoided through post-failover rejoins. These and otherreplication related techniques are described in more detail below.

In some embodiments, the fault tolerant failover protocol of thereplicated state machine (distributed database) in the systems describedherein may include various mechanisms for synchronizing the read/writequorum. In some embodiments, the failover may include a ‘stategathering’ phase. During this step, the read quorum may be filled out ina manner that ensures that everything that satisfies the write quorumwill be found (e.g., user data writes). Note that the read quorum mayalso be referred to as the ‘failover quorum’ since it is the requiredquorum for proceeding with a failover sync-up.

The replication and failover processes described herein may beimplemented by various modules and/or sub-modules of the storage nodeinstances in the system. For example, a log manager may manage the statemachine for updates that are in the process of being replicated. Asnoted above, in some embodiments, the system may implement a singlemaster log shipping based replication approach. In some suchembodiments, updates may begin as log records. These log records may bereplicated in the system, and then (once they are geographicallydurable) they may be committed and later applied to the schema. This maybe thought of as a replication stream in which all replicated updatesare serialized through the stages in strictly increasing order(according to associated log sequence numbers). In some embodiments, thestate machine may track the latest log sequence number (or the logrecord that includes the latest log sequence number) to reach each ofthe states rather than tracking each log sequence number (or itsassociated log record) individually. The state machine may also allowfor batching, and may not be concerned with missed state notifications,since (for example) if a log record with log sequence number L reachesstate S, this always implies that all log records with log sequencenumbers less than L have also reached state S.

As used herein, the following sequence terminology may be assumed:

-   -   Strictly Increasing: this term refers to a sequence that is        always increasing, i.e. a sequence in which every new instance        of the sequence has a higher value than the previous instance.    -   Monotonically Increasing: this term refers to a sequence that is        never decreasing, i.e. a sequence in which every new instance of        the sequence has an equal or higher value than the previous        instance.    -   Dense: this term refers to a sequence that does not contain        holes, i.e. there are no missing members of the sequence. For        example, 1, 2, 3, 4, 5, . . .    -   Sparse: this term refers to a sequence that may contain holes,        possibly a large number of holes (some of which may be large).        For example, 1, 2, 7, 9, 1000, . . .

Various replication and failover techniques may be described hereinusing some or all of the following terms:

-   -   LSN: “Log Sequence Number”. In various embodiments, an LSN may        include a sequence number, an indication of a master epoch, and        a lock generation identifier. These values for a given LSN may        in some cases be denoted by sequence(LSN), epoch(LSN), and        lock(LSN), respectively.    -   LSN Sequence: As used herein, this term may refer to the dense,        strictly increasing integer sequence of LSNs that define the log        stream. This sequence may be defined solely by the sequence(LSN)        of each log record. In some embodiments, each partition may have        one LSN sequence that defines the order of events for its        replicated state machine.    -   Log Stream: As used herein, this term may refer to the stream of        events defined by the LSN Sequence. In some embodiments, outside        of the split operation, there may be only one valid log stream        such that if sequence(LSN₁)=sequence(LSN₂), then        epoch(LSN₁)=epoch(LSN₂) and lock(LSN₁)=lock(LSN₂). In some        embodiments, if this is not true for two LSNs with the same        sequence number, then only one of those LSNs exists in the valid        log stream. In such embodiments, the other LSN exists in an        invalid stream branch that must be snipped out of existence        (e.g., during failover). In some embodiments, the LSN in the        invalid stream branch cannot possibly have been committed based        on the system's quorum semantics.    -   Stream Branch: As used herein, this term may refer to a point in        the Log Stream where two LSNs succeed the previous LSN, which        may also be referred to as a branching point in the log stream.        These two LSNs may have the same sequence number, but may differ        in their lock generation identifiers, and (in some cases) in        their epoch identifiers. Following these two LSNs there may be        two LSN sequences defining two log streams. Each of these log        streams may be referred to as a “stream branch”. In some        embodiments, only one of these branches will survive to become        part of the final committed Log Stream. As described herein, a        branching point may be created by a failover operation that does        not find one or more uncommitted LSNs, where those uncommitted        LSN sequence numbers are redefined by the “just failed over to”        new master replica.    -   Master Replica: As used herein, this term may refer to the        replica that defines (i.e. creates) new LSNs. In some        embodiments, there may always be zero or one master replica at        any given time, and this may be guaranteed by the requirement        that a master replica must hold an external advisory lock.    -   Master Reign: A master replica may act as (i.e. perform the role        of) the master for its replica group until it loses or releases        the external lock. A single master reign may be defined from the        time when the master replica becomes master (i.e. the time when        a replica assumes the role of master replica for its replica        group) until the time that it loses, or otherwise releases, the        external lock that allows its mastership. During the master        reign, the LSN epoch and the LSN lock may remain fixed.    -   LSN Epoch: In some embodiments, this term may refer to the        dense, strictly increasing integer sequence of master reigns        (where a reign is the time during which a replica serves as the        master replica). In some embodiments, when a replica becomes        master, the epoch is increased by one, and the first LSN        produced by the new master replica may be marked as an epoch        change LSN. In the LSN sequence, the epoch may be monotonically        increasing.    -   LSN Lock: In some embodiments, a master replica must have a        single valid lock while serving a reign as master (including        while performing the failover steps to become master). The LSN        lock may be a unique integer associated with a single lock        acquisition. In some embodiments, only one reign as master (i.e.        one epoch) may be associated with a single lock generation. Lock        generation identifiers for a single lock may in some embodiments        comprise a sparsely increasing sequence. In some embodiments, an        external lock manager or lock service may generate the locks,        and the value of lock(LSN) may represent the time of the lock        acquisition transaction.    -   Epoch Change LSN: In some embodiments, the first LSN of a brand        new master epoch may be marked as an “epoch change LSN”. This        convention may be utilized during log conflict detection to        distinguish seemingly valid epoch changes from invalid branches.    -   Valid LSN: As used herein, this term may refer to an LSN that is        not superseded by another LSN in a different branch.    -   Committed LSN: As used herein, this term may refer to an LSN        that is guaranteed to survive. Note that in some embodiments, a        committed LSN may never become invalid. In the replication        protocol described herein, an LSN may be committed when it (or        an LSN that follows it on the same branch) becomes durable while        no other LSN in existence has higher credentials (e.g., a        greater LSN lock value).    -   Invalid LSN: As used herein, this term may refer to an LSN that        is superseded by another LSN in a different branch. In some        embodiments, an LSN may become invalid at precisely the moment        that another LSN with the same sequence but a different lock        value becomes committed. Note that, in some embodiments, an        Invalid LSN will never become committed, and may be doomed to        eventually be snipped. In some embodiments, during log conflict        detection, if two LSNs have the same sequence, the LSN with the        higher lock value may be taken as the valid LSN, and the LSN        with the lower lock value may be considered invalid. Note that        in some embodiments, LSNs may only be invalidated by the current        master committing another LSN with the same sequence. In some        embodiments, a master can only commit LSNs that it produced.        Therefore, in some edge case scenarios an LSN may become invalid        by virtue of a later LSN (in sequence) being committed when the        LSN sequence leading up to the newly committed LSN is on a        different stream branch. In these cases, an invalid LSN may have        a higher lock value than the valid LSN of the same sequence.    -   Invalid Branch: As used herein, this term may refer to an LSN        stream that follows the fork of a Stream Branch that contains        only invalid LSNs.    -   Log Snip: In some embodiments, Invalid Branches may always        (eventually) be snipped, leaving only the valid log stream.

As noted above, log records may advance through a series of replicationphases. For example, log records may be submitted, may become flushed,may be committed, and may then be applied (e.g., to the schema). FIG. 12illustrates various states in a log sequence (e.g., on a master or slavereplica in a replica group) while undergoing a replication process,according to one embodiment. In this example, log records are added tothe log stream 1200 on the right and advance through the various statesmoving from right to left. For example, when a log record is firstsubmitted (shown as 1214), it resides only in local memory (e.g., RAM).While replication is in progress (during the period labeled 1202), thelog record moves (at point z) to the flushed state (shown as 1212),after it is flushed to disk. Once the log record has been flushed todisk, it is considered to be locally durable (during the period labeled1204). Subsequently (at point y), the log record is committed. Once inthe committed state (during the period labeled 1210), the log record isconsidered to be quorum durable (shown as 1206). For example, beingquorum durable may include being durable in a pre-determined number ofdata centers (e.g., in one data center or in another pre-determinednumber of data centers). Finally, the log record is written to theschema (at point x), and is considered to be in the applied state (shownas 1208).

In some embodiments, log records (sometimes referred to herein simply as“logs”) may be committed once they provably meet the definition of acommitted log. In some embodiments, the definition of “committed” may bebased on survivability. In other words, a commit of a log record mayintrinsically happen once the log record is ensured to survive. In fact,once a log record meets this definition, it may be effectively committedregardless of whether the software recognizes this fact yet or not. Itis this point in time that matters in the failover protocol, not theactual time at which the master proclaims that the log is committed. Insome embodiments, the intrinsic definition of commit may be thatdescribed below.

In some embodiments, an LSN (log record) may be committed when it or anLSN that follows it on the same branch becomes durable while no otherLSN in existence has higher credentials (e.g., a greater lock value.) Insuch embodiments, a master may commit a log record only when it hasdetermined that this definition has been met for the log. By meetingthis definition, the replication and failover scheme described hereinmay ensure that the log will survive. In some embodiments, the failoverscheme described herein may ensure that under any single fault scenario,any durable log that has the current highest set of credentials isguaranteed to be included in the valid log stream (i.e. in the survivingstream of log records). With this, replication may just need to followone simple golden rule: a master should only commit log records that itproduced during its current master reign. In some embodiments, any logrecord that the master produced during its current master reign willhave the highest set of credentials (e.g., the highest lock value). Insuch embodiments, if the master has determined that the log has becomedurable, it may infer that the failover protocol will ensure itssurvival.

The log replication mechanism described herein may be a two-phase commitscheme (e.g., a 2PC scheme) that requires a quorum of replication group(a.k.a. replica group) members to durably persist the log and reportback to the master before the commit occurs. At that point, a successfulresponse may be returned the caller (or requestor), since the durabilityrequirement has been fulfilled. In various embodiments (e.g., dependingon the strategy for offering consistent operations that is in effect),the success response may occur as soon as durability allows, or it maybe postponed until the data is applied to the schema on the master.

One embodiment of a commit flow 1300 for a log replication mechanism isillustrated in FIG. 13. As illustrated in this example, the replica in areplica group that has assumed the role of master for the replica group(shown as master replica 1302 in FIG. 13) performs different tasks thanthe other replicas in the replica group (shown as peers 1304 in FIG.13). In this example, the commit flow may begin in response to themaster replica (1302) receiving an update request 1306. The master 1302may submit a corresponding log record to local memory (as in 1310), andthen flush the log record (as in 1312). In some embodiments, a logsequence number (or LSN) may be assigned to the log record during thesubmit phase. Note that flushing the log record (as in 1312) may includedurably writing the log record to disk.

As illustrated in FIG. 13, after flushing the log record to disk, themaster (1302) may then send an “Append” message (1320) to one or more ofthe peers (1304). Each peer 1304 may flush the log record to disk (as in1326), and may reply to the master (1302) with a “Flushed” message(1322) indicating that the log record has been flushed by that peer(1304). If the master (1302) receives “Flushed” messages (1322) from anappropriate number of peers 1304 (e.g., from enough peers in enough datacenters) to satisfy the write quorum for the system (shown as 1314), themaster may return an indication to the user who requested the updatethat the update is considered durable (shown as 1308).

Once the log record is considered durable (e.g., when a quorum of“Flushed” responses indicating that the log has been replicated isreached), the master (1302) may commit the log record (as in 1316), andmay send a “Commit” message (1324) to the peers (e.g., to all of thereplicas in the replica group or to the replicas making up the quorum)to inform them that the log record is durable and can be committedand/or applied. The master (1302) may then apply the log record (as in1318) to the schema, at which point it is reflected in (and discoverablein) the schema (as indicated at 1340). After receiving the “Commit”message (1324) from the master (1302), a peer (1304) may commit the logrecord (as in 1328) and apply the log (as in 1330). Note that applyingthe log may include applying updated data to the schema, and that thismay be done as a background process, in some embodiments.

In some embodiments, a module or system component that is configured toimplement the replication and failover processes described herein (e.g.,module 275 shown within storage node instance 160 in FIG. 2C) mayinclude a collection of sub-modules, each of which performs differentones of these processes. For example, a RequestManager sub-module maycoordinate between a QueryUpdateManager/ChangeManager component and aLogManager component locally, and also between a master replica andremote replica nodes. In some embodiments, a ChangeManager sub-modulemay be responsible for all disk operations, while a QueryUpdateManagermay receive the initial update request. The RequestManager may be passedthe request, and may handle all replication communication. In someembodiments, a LogManager may manage the state machine for log entriesas they move through the “submitted”, “flushed”, and “committed” states.

FIG. 14 illustrates a data replication flow from the perspective of amaster replica for a replica group, according to one embodiment. In thisexample, a component of the master replica that implements both a QueryUpdate Manager and Change Manager (shown as 1402) may receive an updatefrom a requestor (shown as 1410). In various embodiments, this updatemay represent a data update, a membership update, or an updateindicating another special state or operation to be performed. Inresponse, the component 1402 may invoke an update method (at 1428) of aRequest Manager (1404) of the master replica. The Request Manager (1404)may invoke a method of a Log Manager (1406) of the master replica (shownas 1430) to submit a corresponding log record. The Log Manager (1430)may assign an LSN to the log record (at 1444), and send an “Append”message to a Slave Replica (1408), which may be one of two or more otherreplicas in the replica group.

Meanwhile (in this example), at any time after the log record has beensubmitted and its LSN assigned, the Log Manager (1406) may invoke amethod of the Query Update Manager/Change Manager (1402) to flush thelog record (shown as 1432). The Query Update Manager/Change Manager(1402) may then place the log record in a queue for subsequent flushing(as in 1412). For example, a flush-log worker (e.g., a workflow that isconfigured to flush log records and that is executing on the masterreplica and/or for the benefit of the master replica) may be configuredto retrieve the log record from the queue and flush the log record (asin 1414) to disk (as in 1424). Once the log record has been flushed todisk, the Query Update Manager/Change Manager (1402) may invoke a methodof the Request Manager (as in 1434) that is configured to indicate thatthe master replica has flushed the log record to disk. The Slave Replica(1408) may also generate and send a “Flushed” message (1436) back to theRequest Manager (1404) indicating that it has also flushed the logrecord to disk. The Request Manager (1404) may then invoke a method (at1438) of the Log Manager (1406) to indicate that the log record has beenflushed.

As illustrated in this example, once a quorum of the replicas in thereplica group have indicated that they have flushed the log record (asin 1450), the Request Manager (1404) may invoke a commit method (as in1440) of the Log Manager (1406), and the Log Manager (1406) may send a“Commit” message (1448) to the Slave Replica (1408) indicating that thelog record can be committed and/or applied to the schema. The LogManager (1406) may also invoke a method (1442) of the Query UpdateManager/Change Manager (1402) that is configured to apply the log recordto the schema on the master replica. In some embodiments, invoking thismethod may cause the Query Update Manager/Change Manager (1402) toreturn a response to the requestor (as in 1416) and/or to add the logrecord to a queue for subsequent application to the schema (shown as1418). Thereafter, an apply-log worker (e.g., a workflow that isconfigured to apply log records to the schema and that is executing onthe master replica and/or for the benefit of the master replica) may beemployed to apply the log record to the schema (as in 1420). In someembodiments, an additional (or alternate) response may be provided tothe requestor once the log record has been applied to the schema (shownas 1422), e.g., indicating that the requested update has been made.

Note that in some embodiments, the timing of the response to the requestrouter may be based on a “data access” layer policy. Such a policy maybe dependent on how the system has implemented consistent reads and/orother modes of efficiency (which may be based on user request patternsor instructions). In various embodiments, the response may happenimmediately after the commit, or not until after the correspondingupdate is applied in the system.

FIG. 15 illustrates a data replication flow from the perspective of aslave replica in a replica group, according to one embodiment. In thisexample, a Request Manager (1504) of the slave replica may receive an“Append” message (1510) from the Master replica (1502) of a replicagroup to which it belongs. For example, the “Append” message (1510) mayinclude a log record to be appended to the log stream on the slavereplica. In various embodiments, this log record may represent a dataupdate, a membership update, or an update indicating another specialstate or operation to be performed. In response, the Request Manager(1504) may invoke a method of a Log Manager (1506) of the slave replica(shown as 1516). The Log Manager (1506) may then invoke a method of aQuery Update Manager/Change Manager (1508) of the slave replica (shownas 1524) and the Query Update Manager/Change Manager (1508) may placethe log record in a queue (as in 1528), after which a flush-log worker(e.g., a workflow that is configured to flush log records and that isexecuting on the slave replica and/or for the benefit of the slavereplica) may be configured to retrieve the log record from the queue andflush the log record (as in 1530) to disk (as in 1536).

Once the log record has been flushed to disk, the Query UpdateManager/Change Manager (1508) may invoke a method of the Request Manager(as in 1518) that is configured to indicate that the slave replica hasflushed the log record to disk. The Request Manager (1504) may generateand send a “Flushed” message (1512) back to the Master replica (1502)indicating that the slave replica has flushed the log record to disk.The Request Manager (1504) may then invoke a method (at 1520) of the LogManager (1506) to indicate that the log record has been flushed.

At some point (assuming the log record achieves durability at theappropriate number of replicas and/or data centers), the Request manager(1504) of the slave replica may receive a “Commit” message (shown as1514) from the Master replica (1502) indicating that the log record canbe committed and/or applied to the schema (e.g., if the applicable writequorum has been reached). In response to receiving the “Commit” message,the Request manager (1504) may invoke a method of the Log Manager (1506)that is configured to commit the log record (shown as 1522). The LogManager (1506) may then invoke a method of the Query UpdateManager/Change Manager (1508) to apply the log record to the schema(shown as 1526). The Query Update Manager/Change Manager (1508) may thenplace the log record in a queue for subsequent application (as in 1532).As illustrated in this example, an apply-log worker (e.g., a workflowthat is configured to apply log records to the schema and that isexecuting on the master replica and/or for the benefit of the masterreplica) may be employed (as in 1534) to apply the log record to theschema (as in 1538).

In some embodiments, the replication failover protocol may be designedto be fault-tolerant with a geographically aware durability requirement,and may support online membership changes, replication group splitting,and/or geographic replica migration. As previously noted, the system mayutilize a single-master log shipping replication scheme that uses adata-center-centric quorum scheme. The quorum scheme described hereinmay ensure that all updates that could possibly have been reported backto the user as being successful will be found during a failover. Thewrite quorum logic may be implemented in a WriteQuorum class, whileanother class may implement the read quorum (which may also be referredto as the ‘failover’ quorum).

In some embodiments, the quorum logic may be implemented according tothe following criteria:

-   -   A replication group may be defined to exist in N data centers.    -   A write may be considered durable if it has been persisted in K        data centers, where K<=N.    -   A failover may be guaranteed to find all durable writes if and        only if the new master syncs with all members in each of N-K+1        data centers.    -   In some embodiments, the system may be configured with N=3 and        K=2.        According to these assumptions, all writes can be found, and        failover can succeed following any single replica failure        provided that replicas exist in all N data centers prior to the        failure.

One embodiment of a method for replicating a write operation in a datastorage system is illustrated by the flow diagram in FIG. 16. Asillustrated in this example, the method may include receiving a writerequest from a client and routing that write request to the masterreplica of the appropriate replica group (as in 1600). For example, inresponse to a client issuing a write operation using a “PutItem” API, a“DeleteItem” API, an “UpdateItem” API, or any other mechanism forinitiating a write operation (i.e. a state modifying or data modifyingoperation) targeting the data storage system, a write operation may beinitiated by the master replica. The method may include the masterreplica in the replica group shipping a log record for the writeoperation to all members of the replica group as an “append” message, asin 1610. The method may include a slave replica in the replica groupreceiving the log record, appending it to its log, and returning a“flushed” message back to the master replica, as in 1620.

If the write quorum is not reached, shown as the negative exit from1630, the operations illustrated as 1620-1630 may be repeated until awrite quorum is reached, in this example. For example, other slavereplicas may receive the log record, append it to their logs, and return“flushed” messages back to the master replica, and the master replicamay continue to monitor those messages until a quorum is reached (e.g.,until at least one replica from each of K data centers responds with a“flushed” message). Once a write quorum is achieved (shown as thepositive exit from 1630), the master replica may consider the logcommitted, as in 1640. The master replica may then return a response tothe requestor of the write operation (i.e. the client, in this example),and may ship a “commit” message to the other replicas in the group. Asillustrated in this example, the master replica and the other replicasin the group may then apply the write operation indicated in thecommitted log record to the data they manage, as in 1650. Note that inother embodiments, the operations illustrated at 1620-1630 may not berepeated indefinitely in an attempt to meet a write quorum, but theattempt may be abandoned if a timeout expires prior to establishing aquorum or once it is clear that there are not enough replicas remainingin the replica group for a quorum to be established. In suchembodiments, if the write quorum is not reached, the system may notreturn a response to the client, and the client may (or may not)re-issue the write operation.

One embodiment of a method for performing a read operation in a datastorage system is illustrated by the flow diagram in FIG. 17A. In thisexample, the data storage system uses a quorum mechanism for performingread operations. As illustrated at 1710, in this example, the method mayinclude a master replica in a replica group receiving a request toperform a read operation, and (in response) shipping the request to allmembers of the replica group. For example, a client may initiate a readoperation using a “GetItem” or “GetItems” API, or using anothermechanism to initiate the retrieval of data or state stored in the datastorage system. The method may include a slave replica in the replicagroup receiving the request and returning the requested data, as in1715.

If the read quorum is not reached and (in this example) if a timeoutperiod has not yet expired, shown as the negative exit from 1720 and thenegative exit from 1730, the operations illustrated as 1715-1730 may berepeated until a read quorum is reached. For example, other slavereplicas may receive the read request and return the requested data backto the master replica, and the master replica may continue to monitorthose responses until a quorum is reached (e.g., until a result isreturned by all replicas of at least N-K+1 data centers, at which pointthe returned result that is associated with the highest credentials maybe considered the correct result), or until the timeout period expires(shown as the positive exit from 1730). When and if a read quorum isachieved (shown as the positive exit from 1720), the master replica mayreturn a response to the requestor of the read operation, as in 1725. Ifa read quorum is not reached before the timeout period expires (shown asthe positive exit from 1730), the method may include the master replicareturning an error message to the requestor, as in 1735.

Another embodiment of a method for performing a read operation in a datastorage system is illustrated by the flow diagram in FIG. 17B. In thisexample, the data storage system does not use a quorum mechanism forperforming read operations. As illustrated in this example, the methodmay include receiving a request for a read operation from a client, asin 1740. For example, a client may initiate a read operation using a“GetItem” or “GetItems” API, or using another mechanism to initiate theretrieval of data or state stored in the data storage system. If theread is to be performed as a consistent read operation (shown as thepositive exit from 1745), the method may include routing the request tothe master replica for the appropriate replica group, as in 1750. Inthis case, the master replica for the replica group may receive therequest and return the requested data to the client, as in 1755. Forexample, in some embodiments, the master replica maintains an item cachestoring information about items (or logs) that have been committed up tothe current point. Therefore, the most recent version of the requesteddata may be present in that cache and/or on disk, and master may serveit without consulting any other replicas. Note that a read operation maybe performed as a consistent read operation if the underlying systemimplements consistent read operations for all read operations or if thisoption is specified for the partitions hosted on the replica or for therequested read operation itself, in different embodiments.

If the read operation is not to be performed as a consistent readoperation (shown as the negative exit from 1745), it may be performed asan eventually consistent read operation. In general, an eventuallyconsistent read may be served by any replica in the appropriate replicagroup. As illustrated in this example, the request may be routed to anarbitrary replica in the appropriate replica group, as in 1760, and thatreplica may receive the request and return the requested data to theclient, as in 1765. Note that a read operation may be performed as aneventually consistent read operation if the underlying system implementseventually consistent read operations for all read operations or if thisoption is specified for the partitions hosted on the replica or for therequested read operation itself, in different embodiments.

In some embodiments, instances of a MasterContext class may hold and/oradvance the master state machine state. In other words, theseMasterContext instances may implement all failover logic that drives thestate machine. The state machine may be driven by attempts to becomemaster of a replica group once the external lock for the replica groupis held. A replica acting as the master replica may transition back toslave status when the lock is lost, or if the node cannot otherwiseperform its role as the master replica.

A failover process may be performed in response to various system eventsor conditions (e.g., in response to the failure of a storage nodeinstance or communication link thereto, or in response to a change inpartitioning or replica group membership). Failover may be driven by oneof the replicas in a replica group attempting to become the master forthe group. Failover may be considered complete when the replicaattempting to become master assumes mastership of the replica group bysuccessfully completing all of the steps required to become master.

In some embodiments, the failover protocol may be defined by thefollowing series of steps, which are described in more detail below:

-   -   1. Acquire External Advisory Lock    -   2. Gather State    -   3. Fetch Tail    -   4. Replicate Tail    -   5. Write the first LSN of the new epoch    -   6. Wait for the epoch change LSN to become durable, then commit        the tail    -   7. Assume master

In some embodiments, in order for a replica to become the master for itsreplica group, it must first acquire an external advisory lockdesignated for the replication group. In some embodiments, this lock maybe uniquely identified by the partition identifier of the data partitionfor which the replication group manages data. Note that in someembodiments, only one replica may be able to hold the lock at any onetime (e.g., an external lock manager or service may ensure that this isthe case). In some embodiments, only one master attempt may be made perlock acquisition. Each lock acquisition may include generation and/orassignment of a unique lock generation identifier, which may beguaranteed to be greater than the identifier of any previous lockacquisition (e.g., the external lock manager or service may ensure thatthis is the case). In some embodiments, a replica that successfullyacquires the lock may drive the failover until it succeeds in becomingmaster, or until it fails (in which case it may release the lock so thatanother replica may attempt to become master for the replica group).

In some embodiments, gathering state may include querying all members ofa replication group for their latest flushed and committed LSNs, and fortheir membership version (as described below). While gathering state, atest for a quorum may be performed upon receipt of every response tothose queries. In some embodiments, peers are counted for the quorum(i.e. included in the quorum) if and only if they indicate that theysupport the candidate peer for master of the replication group. In someembodiments, mastership is not supported by a peer if it knows of (e.g.,if it has observed) a newer membership version (which may requirecatch-up and re-iteration), if it does not host the replica (which mayindirectly indicate that the membership version is out of date), or ifit has observed a newer lock value (as a safe-guard).

In various embodiments, synchronization for changing the quorum set(i.e. the set of participants in the quorum scheme) may utilize a‘membership version’ (or more generically a ‘quorum version’) that isupdated through a replicated change, and whose current value ismaintained for the replica group in a membership version indicator(e.g., in metadata maintained by the master replica). In someembodiments, each of the other replicas may maintain a membershipversion indicator that stores the most recent membership version ofwhich it is aware (i.e. that is has observed). In some embodiments, areplica that is attempting to become master may iterate on filling outthe failover quorum (i.e. the read quorum) itself whenever a higherquorum version is discovered. For example, on a failover, the replicaattempting to become the new master may fill out the failover quorum.Upon discovering a higher quorum version, the replica attempting tobecome the new master may immediately synchronize the data leading up tothe new version. After synchronizing, the replica may be ensured that ithas found everything that used the pre-quorum change quorum fordurability (in other words, it used the correct failover quorum fordiscovery of that data). The replica may then start gathering stateagain (filling out the failover quorum) using the newly defined quorum.

In some embodiments, the master hopeful must catch up its membershipversion during this step if any participating peer knows of (andreports) a more recent membership version. For example, the replica maybe required to catch up (synchronize) its log to the more recentmembership change, and then start the gathering state step over againusing the new membership. Note that this may iterate more than once, insome cases.

Note that if the replica's log is snipped due to detecting a logconflict during this catch-up, it may iterate without reaching themembership change (and may be forced to catch up from farther behind).Note that if any member reports a newer membership version, and thereplica that is attempting to become master is no longer part of themembership, the attempting replica may abandon the attempt, and may stophosting the replica. Note that in some embodiments, only log recordsthat are known to be in conflict (i.e. that have been determined to bein conflict) are snipped from the log and then further iterations of thecatch-up operation may be performed in order to synchronize the log(e.g. unwinding the log stream as any additional conflicts arediscovered during these additional iterations).

In some embodiments, the failover protocol described herein may enablesafe membership changes (with respect to the quorum scheme), and mayallow the quorum to be safely reconfigured when necessary. Sincereplication group splitting is also (in part) a membership change, itmay be one of the primitives that enable partition splitting as well.Once mastership is supported by the failover quorum of peers, thereplica may move to the next step.

In some embodiments, fetching the tail may include fetching any missingtail of the log (any log records not already appended to the replica'slog) from the quorum member with the highest LSN credentials. In suchembodiments, the highest LSN credentials may be the credentials thatinclude the highest lock value, or the credentials that include thehighest sequence value (e.g., if the lock values of multiple log recordsare the same). Again note that if the replica's log is snipped due todetection of a log conflict during a process to catch-up the log tail,the replica may iterate, starting the catch-up again from the point atwhich the conflicting log record was snipped.

In some embodiments, once the tail is caught up, the replica mayreplicate the tail such that the tail meets the durability requirement(e.g., that it is durable in at least K data centers). In someembodiments, the tail may be replicated to all nodes, but only theminimum necessary durability may be required.

Once the tail is verified replicated, the replica may write the firstLSN of the new epoch. In other words, the replica may write metadataabout the change in the mastership for the replica group, and thecorresponding LSN may be marked as an “epoch change” (which may beuseful later for log conflict detection and determining snip log cases).As mentioned above, in some embodiments, the master is not allowed tocommit log records that it did not produce. In some embodiments, inorder to avoid moving forward until it is certain that the log is fullyresolved, after writing the first LSN of the new epoch, the replica maywait for the epoch change LSN to become durable, and then may commit thetail, and flip to being the master.

In some embodiments, the fact that the master hopeful stays in the‘assuming master’ state until the epoch LSN is durable may prevent newuser updates from being accepted prior to the replica committing thetail. Once the epoch LSN is durable, there can be no conflicts with thefound tail in a subsequent failover. A this point, the log stream hasbeen completely resolved up to and including the new master epoch LSN.

In some embodiments, once all of the steps described above have beencompleted, the replica may be cleared to become the new master. Notethat in some embodiments, any or all of these failover steps may betimed. In such embodiments, each step (or iteration of a step) may beallotted a maximum time in which it may run. In some embodiments, thesetimeouts may be updated dynamically (e.g., they may be reset based onbatched progress and iteration). In some embodiments, the system mayinclude safeguards against data causing sudden timeout failures thatwould be persistent.

One embodiment of a method for performing a replication failover processin a data storage system is illustrated by the flow diagram in FIG. 18.As illustrated in this example, the method may include a replica in areplica group initiating an attempt to become the master for the replicagroup, as in 1810. The method may include the replica that is attemptingto become the master replica acquiring the external lock associated withthe replica group and/or with the data partition it manages, as in 1820.As described above, the method may include the replica that isattempting to become the master replica gathering state information fromanother replica in the replica group, as in 1830.

As illustrated in this example, if the other replica supports thismastership attempt (shown as the positive exit from 1840), the methodmay include the adding the other replica to the failover quorum, as in1850. On the other hand, if the other replica does not support thismastership attempt, the other replica is not added to the failoverquorum. This is illustrated in FIG. 18 by the feedback from the negativeexit of 1840 to 1830. As illustrated in this example, the replicaattempting to become the master replica may continue gathering stateinformation from other replicas in the replica group until the failoverquorum is reached. This is illustrated in FIG. 18 by the feedback fromthe negative exit of 1860 to 1830. In other embodiments, rather thanwaiting indefinitely until the failover quorum is reached, theseoperations may only be repeated until a timeout period expires or untilit is clear that there are not enough replicas remaining (i.e. as yetnon-reporting) to reach the failover quorum. Note that replicas that arenot included in the failover quorum may end up with an invalid branch ofthe log stream if they have flushed log records that were not found inthe failover quorum and are thus superseded by log records produced bythe newly elected master (assuming the replica succeeds in assuming therole of master replica).

Once the failover quorum is met, shown as the positive exit from 1860(e.g., once the replica attempting to become the master replica for thegroup gathers information indicating that all replicas from N−K+1 datacenters support this mastership attempt), the failover process maycontinue. In this example, the method may include the replica that isattempting to become the master replica for the replica group fetchingany missing tail of the log stream that is found within one of the otherreplicas in the quorum, as in 1870. If such a tail is found, the methodmay include the replica that is attempting to become the master replicacatching up to the tail and replicating it (e.g., sending its logrecords to the other replicas in the group in one or more “append”messages). The replica attempting to become the master may then writethe first log record of a new epoch (an epoch in which it is the masterreplica), as in 1880. When this log record is durable (e.g., when thisepoch change log record has been successfully replicated within thereplica group and the write quorum is met with durability in at least Kdata centers), the replica attempting to become the master replica maycommit the now-replicated tail. As illustrated in this example, thereplica that is attempting to become the master may at that point assumemastership for replica group, as in 1890.

One embodiment of a method for acquiring an external lock for a replicagroup (as in 1820 of FIG. 18) is illustrated by the flow diagram in FIG.19. As illustrated in this example, the method may include one or morereplicas in a replica group expressing interest in an external lockdesignated for the replica group (which may indicate their intention toattempt to assume the role of master replica for the replica group), asin 1910. The method may include one of the replicas in the replica groupattempting to acquire the external lock that is designated for thereplica group and/or for a data partition it hosts (e.g., during afailover process), as in 1920. In various embodiments, such a lock maybe maintained and/or managed on behalf of the replica or partition by anexternal lock manager or an external lock service. If the external lockmanager or service does not grant the lock to the replica that isattempting to become the lock manager (shown as the negative exit from1930), the method may include the same replica or another replica in thereplica group attempting to acquire the external lock designated for thereplica group, shown as the feedback from 1930 to 1920.

If the external lock manager or service does grant the lock to thereplica that is attempting to become the lock holder or owner (shown asthe positive exit from 1930), the method may include the lock manager orservice assigning a unique lock generation identifier for the lock, asin 1940. In some embodiments, other credentials may also be assigned bythe lock manager or service (as described herein). The failover processmay then continue. If the replica that acquires the external locksucceeds in becoming the master replica for the replica group, shown asthe positive exit from 1950, the failover process may be complete, as in1960. For example, the replica that acquires the external lock may havegathered state from the other replicas in its replica group in anattempt to build a quorum to support its mastership attempt, and thatattempt may have been successful. If the replica that acquires theexternal lock does not succeed in becoming the master replica for thereplica group, shown as the negative exit from 1950, that replica mayrelease the external lock, as in 1970. Subsequently, the replica mayagain attempt to acquire the external lock or another replica mayattempt to acquire the external lock. This is illustrated in FIG. 19 bythe feedback from 1970 to 1920.

One embodiment of a method for filling out a failover quorum isillustrated by the flow diagram in FIG. 20. As illustrated in thisexample, the method may include a replica that is attempting to becomethe master replica for its replica group beginning to gathering stateinformation from other members of the replica group, as in 2010. Forexample, the replica that is attempting to become the master replica mayquery a peer for its latest flushed and committed log records, and itsmembership version, as in 2015. The methods may include determining(based on the information gathered) whether the peer knows of (e.g., hasobserved) a newer membership version than the newest membership versionthat is known to the replica that is attempting to become the masterreplica, as in 2020.

If the peer knows of a membership version that is newer than the newestmembership version that is known to the replica that is attempting tobecome the master replica, shown as the positive exit from 2020, themethod may include the replica attempting to catch up to the newermembership change, as in 2025. Such a catch-up operation is described inmore detail herein. If the catch-up operation is not successful, shownas the negative exit from 2030, the method may include the replicaabandoning its attempt to become the master replica, as in 2075. If thecatch-up operation is successful, shown as the positive exit from 2030,and if the replica is still a member of the replica group (according tothe newer version of its membership), shown as the positive exit from2035, the method may include the replica re-starting the state gatheringoperation with the new membership, as in 2040. If the catch-up operationis successful, shown as the positive exit from 2030, but the replica isno longer a member of the replica group (according to the newermembership version), shown as the negative exit from 2035, the methodmay include the replica abandoning its attempt to become the masterreplica, and no longer hosting the replica, as in 2045.

If the peer does not know of (e.g., has not observed) a membershipversion that is newer than the newest membership version that is knownto the replica that is attempting to become the master replica (shown asthe negative exit from 2020), if the peer hosts the replica (shown asthe positive exit from 2050), and if the peer has not seen a greaterlock value than the replica has seen (shown as the negative exit from2055), the method may include the replica that is attempting to becomethe master for the replica group including the peer in the failoverquorum, as in 2060. Otherwise (e.g., if the peer does not host thereplica and/or if the peer has seen a greater lock value than thereplica has seen), the method may include the replica that is attemptingto become the master querying one or more other replicas in the replicagroup. This is illustrated in FIG. 20 by the feedback from the negativeexit of 2050 or the positive exit of 2055 to 2015. If the failoverquorum is not met (shown as the negative exit from 2065), the method mayinclude repeating the state gathering operations beginning at 2015 untilthe quorum met or until a timeout period has expired (not shown). If thefailover quorum is met (shown as the positive exit from 2065), themethod may include continuing the failover process, as in 2070.

One embodiment of a method for performing a catch-up operation on a logstream tail (e.g., as shown in element 1870 of FIG. 18) is illustratedby the flow diagram in FIG. 21. As illustrated at 2110, in this example,the method may include a replica that is attempting to assume mastershipof its replica group beginning an operation to catch-up to any missingtail of the log. The method may include determining whether anotherreplica in the failover quorum includes a log record with the highestlock value of any log records in the log stream, as in 2120, and/ordetermining whether the other replica includes a log record with thehighest sequence value among those with the same highest lock value, asin 2130. If so (shown as the positive exit from 2120 or 2130), themethod may include the replica that is attempting to assume mastershipfetching the tail of the log from the other replica, as in 2140. Themethod may also include the replica that is attempting to assumemastership attempting to catch-up to the tail of the log from the otherreplica, as in 2150. For example, the replica may attempt to synchronizeup to the tail by appending any missing log records (up to the logrecord with the highest credentials) to its log.

As illustrated in this example, the method may include determining(while attempting to synchronize up to the tail) whether there is a logconflict that causes the replica's log to be snipped, as in 2160. If so,the replica's log may be snipped (e.g., the log records that are knownto be in conflict may be deleted) and the replica may iterate on theprocess one or more times, each time starting the catch-up operationagain from the point in the log prior to the latest detected conflict.This is illustrated in FIG. 20 by the feedback from the positive exit of2160 to 2140. If no conflict that causes the replica's log to be snippedis detected, shown as the negative exit from 2160 (e.g., if the replicahas successfully caught up to the tail of the log), the method mayinclude replicating the tail of the log to all of the other replicas inthe replica group (or at least to the replicas included in the failoverquorum), and verifying the minimum required durability of the logrecords in the tail of the log, as in 2170.

If the other replica does not include a log record with the highest lockvalue of any log records in the log stream or a log record with thehighest sequence value among those with the same highest lock value(shown as the negative exits from 2120 and 2130), and there are no morereplicas in the failover quorum (shown as the negative exit from 2180),there may be no missing tail of the log stream (e.g., no log tail havinglog records with higher credentials than the log records in the logstream of the replica attempting to assume mastership), as in 2190. Ifthe other replica does not include a log record with the highest lockvalue of any log records in the log stream or a log record with thehighest sequence value among those with the same highest lock value(shown as the negative exits from 2120 and 2130), but there are morereplicas in the failover quorum (shown as the positive exit from 2180),the method may include continuing to look for a missing tail thatincludes log records with the highest credentials of any log records inthe log stream. This is illustrated by the feedback from the positiveexit of 2180 to 2120.

As previously noted, in some embodiments, replication groups (or replicagroups) may be managed through replicated metadata changes. In suchembodiments, when any member or members are added or removed, or whenthere is a change in the state of a member or members (or anycombination of these operations), these membership change operationsthemselves may be performed as replicated changes. In general, anymetadata that affects the replication group as a whole may be managedthis way.

As previously noted, synchronization for changing the quorum set (i.e.the set of participants in the quorum scheme) may utilize a ‘membershipversion’ (or more generically a ‘quorum version’) that is updatedthrough a replicated change. For example, in some embodiments, ametadata change may be written that increases the quorum versionwhenever a parameter of the quorum is altered (e.g. when a member beginsparticipating in the quorum or is added to the set of quorumparticipants, when a member stops participating in the quorum, or whensome fundamental property of the quorum itself is changed). In someembodiments, the systems described herein may use a locality-basedquorum. In some embodiments, replicated changes may be used to changethe valid data center set, and/or the minimum durability requirement forthe write quorum (which in turn may redefine the read quorum).

In some embodiments, membership changes may synchronize with the quorumaccording to the following criteria. First, as described above,membership itself may be versioned, and all membership changes mayresult in an increase in the membership version. Membership changes maybe implemented as metadata writes in the LSN stream. During failover, ifany member reports a higher membership version during the “gather state”phase (while filling out the failover quorum), the member attempting tobecome master must immediately synchronize its log up to the membershipchange (or alternatively abandon its attempt to become master), and (ifcontinuing) start the “gather state” phase (filling out the failoverquorum) over again using the new membership. By doing this, the systemmay be certain that a) every LSN that became durable under the oldmembership has been found (because it was possible to synchronize thelog all the way up to the membership change), and b) every LSN afterwardused the new membership for durability assessment (up to and includingthe next membership change if there is one). This properly syncsmembership with quorum at failover time (which is when it matters).

Note that the replica might snip the log while synchronizing to themembership change LSN, in which case one or more iterations may beperformed using the old membership. If, after synchronizing, yet anothermembership change is discovered, additional synchronizing and iteratingmay be required. In general, the replica may always need to find thequorum using the correct membership, and so it may need to iterate oneach membership change.

One embodiment of a method for performing a replica group membershipchange is illustrated by the flow diagram in FIG. 22. As illustrated inthis example, the method may include a replica that is acting as themaster replica for a replica group incrementing an indication of amembership version for the replica group, as in 2210. The method mayinclude the replica acting as master shipping a membership change logrecord to the other members of the replica group as a metadata write, asin 2220. The metadata may include the incremented membership versioninformation. Since the membership change is replicated just as any otherwrite operation in the system, it may also be subject to durabilityrequirements (e.g., the same durability requirements as other writeoperations or different durability requirements, in variousembodiments).

As illustrated in this example, if the master replica receives anindication that the replicated membership change is durable (e.g., thatthe applicable write quorum requirements have been met), shown as thepositive exit from 2230, the master replica may send a commit messagefor the membership change to the other replicas in the replica group, asin 2235, and the membership change may be considered committed, as in2260. In response, the membership change may be applied by all of thereplicas in the replica group, as in 2270.

As illustrated in this example, if the master replica does not receivean indication that the replicated membership change is durable (e.g.,that the applicable write quorum requirements have been met for thereplicated membership change), shown as the negative exit from 2230, butthe master replica receives an indication that a later write operationis durable (shown as the positive exit from 2240), the master replicamay send a commit message for the later write operation to the otherreplicas in the group (as in 2245), which may imply that the (earlier)membership change is also committed. Therefore, in response to the laterwrite operation committing, the membership change may be consideredcommitted (as in 2260) and the membership change may be applied by allof the replicas in the replica group (as in 2270). For example, in someembodiments, the master replica may not wait for an indication that thereplicated membership change is durable, since it may not need toprovide an indication of the result of the change to any externalrequestor, but if it is determined that a later write operation hascommitted, the master replica may correctly assume everything before it(include the membership change) must have committed. If the masterreplica does not receive an indication that the membership change isdurable or that a later write is durable (shown as the negative exitsfrom 2230 and 2240), the membership change may not be committed (as in2250). As illustrated in this example, this may be discovered during asubsequent failover process (as a log conflict or invalid branch). Notethat in other embodiments, the method may include the master replicamonitoring the responses received from the other replicas in the replicagroup until it determines that the membership change is durable or untilthe timeout period expires.

One embodiment of a method for synchronizing up to a replica groupmembership change during failover is illustrated by the flow diagram inFIG. 23. As illustrated in this example, the method may include areplica that is attempting to become master for a replica group queryinga peer for its latest flushed and committed log records, and itsmembership version, as in 2310. For example, the replica that isattempting to become master for a replica group may query a peer in itsreplica group (as it is currently known to the replica that isattempting to become master). If the information received from the peerindicates that the peer knows of a newer membership version than thatknown by the replica that is attempting to become master (shown as thepositive exit from 2315), the method may include the replica that isattempting to become master attempting to catch up to the newermembership change that is known to the peer, as in 2320. For example,the replica that is attempting to become master may attempt tosynchronize up to the newer membership change by appending any missinglog records (e.g., log records that are not currently found in its logstream) up to newer membership change to its log stream. If theinformation received from the peer does not indicate that the peer knowsof a newer membership version than that known by the replica that isattempting to become master (shown as the negative exit from 2315), andif state has been gathered from enough peers to establish a quorum(shown as the positive exit from 2365), no catch-up operation(s) may berequired (e.g., on the first iteration of the process), or the necessarycatch-up operation(s) may be complete (e.g., on a subsequent iterationof the process), as in 2370. If the information received from the peerdoes not indicate that the peer knows of a newer membership version thanthat known by the replica that is attempting to become master (shown asthe negative exit from 2315), but state has not been gathered from allof the peers or from at least enough peers to establish a quorum (shownas the negative exit from 2365), the method may include re-starting thestate gathering operation illustrated in FIG. 23 beginning at 2310.

As illustrated in this example, if a log conflict is detected and thelog of the replica that is attempting to become master is snipped beforethe newer membership change (shown as the positive exit from 2325), themethod may include the replica attempting to catch up with the log ofthe peer from farther behind, as in 2330. For example, the method mayinclude repeating the operations illustrated as 2320-2330 from the pointin the log at which a conflicting log record was snipped away. This isillustrated in FIG. 23 as the path from 2330 to 2320. If (on the firstor any other iteration of the operations illustrated at 2320-2330) noconflicts are detected and if the catch-up operation is successful(shown as the negative exit from 2325 and the positive exit from 2335),and if the replica is still a member of the replica group according tothe newer membership change (shown as the positive exit from 2345), themethod may include the replica re-starting the state gathering operationwith the new membership, as in 2360, and the path back to 2310. On theother hand, if (on the first or any other iteration of the operationsillustrated at 2320-2330) no conflicts are detected, but the catch-upoperation is not successful (shown as the negative exit from 2325 andthe negative exit from 2335), the method may include the replicaabandoning its attempt to assume mastership of the replica group, as in2340. However, if the catch-up operation is successful, but the replicais not still a member of the replica group according to the newermembership change (shown as the negative exit from 2345), the method mayinclude the replica abandoning its attempt to assume mastership of thereplica group, and discontinuing hosting the replica, as in 2350.

As illustrated in FIG. 11 and described above, in some embodiments, astorage system may support an operation to split a partition into twonew partitions. Splitting a partition may in some embodiments be afailover-time operation that involves replication group splitting (alsoknown as live-repartitioning or “sharding”). Splitting a partition maybe thought of as a combination of a membership change and an identitychange for the replication group in a single operation. Note that themaster locks (the external locks that identify the replicas that areeligible to be the master for each replica group) are identified bypartition identifiers (which will change due to the split). Therefore,all replica group members that append a split log record may releasetheir interest in the old master lock (the lock for the pre-splitreplica group), and may register an interest in the new lock (i.e. thelock associated with an identifier of the partition to which they switchas a result of the split).

While appending the split operation results in two new replicationgroups, from the perspective of any one replica, the replica undergoes amembership change (as its replica group is reduced in size), and anidentity change (as its partition id and data range change). Since thenew metadata record (which may be referred to as the split partitionrecord, or simply the “split record”) for the new membership changecontains the membership information of both new groups, each new groupis aware of the other's existence. In addition, since the new metadatarecord (the split record) also contains the partition ID of the previousreplica group, each new replica group remembers where it came from(which may be useful in various catch-up scenarios). Since the splitpartition record indicates a membership change, it may synchronize withthe quorum in exactly (or substantially) the same fashion as any othermembership change (as described above).

The replication primitive for splitting the group may utilize the quorumsynchronization mechanism, in some embodiments. As previously noted,this system may use an external advisory lock manager or lock service todetermine which replica has the right to become master, and the lock maybe identified by the partition identifier. Note that the approachdescribed herein may work for all configurations of the locality basedquorum scheme described above when the failover quorums for thepre-split and post-split replica groups overlap each other's writequorums. For example, if a replica group is to be split into two newreplica groups, this approach may be applied when N−K+1>N/2, and the tworeplica groups are spread identically across the same data centers. Insome embodiments, the standard configuration may include values of N=3,K=2, which meet this constraint. Note that (in this example) if theconfiguration is altered yielding N−K+1<=N/2, splits should not beperformed without reconfiguring the quorum. In other words, for a splitto be safe (i.e. guaranteed not to accidentally yield a “split brain”scenario, in which different replicas assume mastership of the pre-splitreplica group and one or more post-split replica groups following theattempted split), the failover quorum must work out to be a simplemajority of the configured data centers. In other words, the failoverquorum must be a set of replicas that satisfies both the failover quorum(i.e. the number of replicas required) and the requirement that thepre-split and post-split replica groups overlap each other's writequorums (i.e. that the pre-split group's failover quorum must overlapthe write quorum of each of the post-split groups, and vice versa).

In one example, it may be assumed that the replication group is grown toN*2 members, including two nodes in each of the N data centers. In thisexample, the split may be performed as follows. First, the masterreplica may be instructed to split the group into two groups, eachcontaining N nodes, including one in each of the N data centers. Next,the master replica may lock the replica exclusively, and may write asplit membership record to the log stream expressed as a quorum change(increasing the quorum version). The split membership record may defineany or all of: the two group memberships, the two groups' post-splitpartition identifiers, the two groups' post-split data ranges, and/orthe original partition identifier. The master may then ship the new logrecord to all members of the pre-split group (e.g., asynchronously, notwaiting for any acknowledgements), after which it may append the logrecord to its log and execute the split operation that is indicated inthe split membership log record. Appending the split membership logrecord may cause the master to: change its partition identifier to theidentifier of the post-split group of which it is a member, change itsdata range to match that of the post-split group of which it is amember, release its master lock (thus demoting it from master status ofthe pre-split partition group to slave status), and express interest inthe external lock associated with its new partition identifier. Aspreviously noted, this mechanism for splitting a partition may beextended to allow splits into more than two new partitions (andcorresponding replica groups), in some embodiments. In such embodiments,in order for the split to succeed, overlapping failover quorum and writequorum requirements described above may need to be met.

Since the split partition record changes the partition identifier forboth new replica groups, it requires each group to failover. Note thatthe slave replicas of the original replica group may or may not receivethe new split record right away. Therefore, at this point, there arepotentially three master elections that may begin, and that may all beattempted at the same time. First, since the master of the originalpartition identifier has given up the external advisory lock for theoriginal replica group, any slave replica that has not yet applied thesplit record may acquire the external lock for the original replicagroup and may attempt to become master for the original partition. Inaddition, any slave replica that has applied the split record and isincluded in the membership for the first new replica group may acquirethe external lock for the first new replica group (and the correspondingfirst new partition identifier) and attempt to become the master for thefirst new replica group. Finally, any slave replica that has applied thesplit record and is included in the membership for the second newreplica group may acquire the external lock for the second new replicagroup (and the corresponding second new partition identifier) andattempt to become master for the second new replica group.

In some embodiments, a slave replica that is attempting to become masterfor the original replica group will meet one of two fates: it willachieve the failover quorum (and become master for the originalpartition), or it will discover (while attempting to fill out thefailover quorum) that the split record exists and be required tosynchronize up to it. Discovering the existence of the split record maycause the replica to abandon its attempt to become master, since it mustnow release the external lock for the original replica group. Thereplica may now recognize the new post-split replica group of which itis a member, and may attempt to acquire the external lock for the newpost-split replica group. If the slave replica succeeds in becomingmaster for the old partition identifier, this means that the splitrecord failed to become durable within the original replica group, andis now destined to be snipped away on any replica that did manage toappend it.

In some embodiments, if the split record achieves minimum durability,any slave replica that acquires the external lock for the original groupwill be guaranteed to find the split record when filling out thefailover quorum (within the configured fault tolerance). In this case,the split may be guaranteed to succeed. However, if the split recorddoes not achieve durability, and a slave replica fills out the failoverquorum without discovering the split record, the slave replica maybecome master for the original replica group, and the slave replica mayredefine the log sequence number that was the split record with its ownepoch change record, or with a subsequent log sequence number thatfollows the log sequence number of the epoch change. In this case,neither post-split replica group will be able to elect a master becauseeach of their failover quorums will overlap the quorum used in electingthe master for the original group. This is because the post-splitreplica groups are guaranteed not to be able to elect a master so longas the quorum configuration satisfies the overlapping quorumrequirements described above. Furthermore, during any attempts of thepost-split replica group members to become master for a post-splitreplica group, the log conflict will be detected, causing the masterhopeful to snip their log (thus snipping away the split record). Thismay result in the replica reverting back to its old state, including itsinterest in the master lock for the original replica group. In otherwords, detecting the log conflict may cause the replica to abandon itsattempt to become master, and register its interest in the originalreplica group's master lock again.

As noted above, if either of the post-split groups succeeds in electinga master, the split may be guaranteed to succeed. This is because theminimum durability requirement for the post-split groups is the same asthe minimum durability requirement for the pre-split group, electing amaster requires at least the minimum durability requirement of replicasto be caught up to (e.g., to have flushed) the epoch change record, andthe epoch change record occurs after the split record in the logsequence. In other words, if either post-split group succeeds inelecting a master, it is guaranteed that the split record was durable inthe pre-split replica group. Therefore, the split itself is guaranteedto succeed. Stated another way, once master election succeeds for eitherof the new replica groups, the split is guaranteed to succeed, sincesuccessfully achieving the failover quorum for either new groupsatisfies the durability requirement for the original replica group. Insome embodiments, this process for splitting a partition works inconjunction with the failover protocol such that either the split doesnot become durable (and is snipped away fairly quickly), or the failoverattempts push the split to completion fairly quickly. In general, thewhole failover process following a split may be performed roughly asquickly as a single failover. Note that in some embodiments, thisapproach may require that the pre-split and post-split replica groupsmeet the quorum overlap requirements described herein in the same datacenters.

Note that catch-up (e.g., hole filling) requests from pre-split nodes topost-split nodes may in various embodiment allow a pre-split node tocatch up in one of two ways. For example, if the split is complete, thepre-split node may need to specify its new partition identifier (whichit may obtain from a heartbeat issued by the new replica group) in itsrequest, and only members of the correct post-split group can fulfillthis request. On the other hand, if the split is not yet consideredcomplete, the post-split nodes may recognize that the request is comingfrom a pre-split node, and may offer up log records up to and includingthe split record. The pre-split node may then (following the appendingof the split record) be required to make further requests for missinglog records to nodes in its post-split group membership using the newpartition identifier for its post-split group.

In some embodiments, heartbeats may carry the partition identifier(s) ofpre-split partitions until the split record becomes flush stable (atwhich point it is certain that there are not any members of thepre-split replica group that are still behind the split). Any pre-splitnodes may note this (e.g., this may be how they recognize that theheartbeat is meant for them), and may use the pre-split partitionidentifiers in subsequent catch-up requests. In some embodiments, asplit may be considered “complete” when both new replica groups havecommitted the split record. Each of the new replica groups may monitorthis independently, and may declare the split complete when it receivesconfirmation that the split record has been committed. At that point, anew membership change record may be propagated to clear the splitrecord, and to write the new membership record that contains only thepost-split group information, in some embodiments. It may not be untilthis point that one of the new groups is allowed to perform anothersplit.

One embodiment of a method for splitting a replicated partition isillustrated by the flow diagram in FIG. 24. As illustrated in thisexample, the method may include (e.g., in response to a request to splita partition) the replica acting as master of a replica in the groupinitiating the addition of one or more replicas in the replica group, asin 2410. The method may include the replica acting as master shippingone or more membership change log records to the other members of thereplica group, each indicating the addition of a replica in the group,as in 2420. In some embodiments, each membership change log record mayinclude an incremented membership version, as described above.

As illustrated in this example, the method may include determiningwhether the membership change(s) are durable and whether the replicas inthe expanded replica group are sufficiently caught up to the most recentlog records (as in 2430). If not, e.g., if the membership change logrecord replication does not meet the applicable write quorumrequirements, the method may include the master replica monitoringresponses from the other replicas in the replica group until it receivessufficient indications that the membership change is durable. In anotherexample, if there are not at least enough replicas in the write quorumthat are caught up to allow master elections to be held, the method mayinclude the master replica monitoring the state of the other replicas(and/or refusing to accept a request to split the replica group) untilat least the minimum number of replicas are caught up. If the masterreplica does receive sufficient indication(s) that the membershipchange(s) are durable and that enough replicas are sufficiently caughtup (shown as the positive exit from 2430), e.g., if the membershipchange log record replication meets the applicable write quorumrequirements, the method may include the replica acting as mastershipping a membership change log record to the other members of thenewly expanded replica group indicating that the expanded replica groupis to split into two new replica groups, as in 2440. In someembodiments, this membership change log record may include informationindicating which replicas are to be included in each of the new replicagroups, a new partition identifier for each of the new replica groups,and a new data range for each of the new replica groups. As previouslynoted, the techniques described herein for splitting a replica group maydepend on the overlapping quorum requirements described above.

In some embodiments, subsequent to a membership change to split areplica group into two new replica groups, a replica in at least one ofthe two new replica groups may attempt to become master of its newreplica group, as in 2450. For example, attempting to become master ofthe new replica group may include gathering state information aboutother replicas in the new group and determining whether they support themastership attempt, as with any other failover operation. In the exampleillustrated in FIG. 24, if no attempt to become master of a new replicagroup is successful (shown as the negative exit from 2460), an attemptby one of the replicas of the expanded replica group to become master ofthe expanded replica group may be successful, as in 2470. If it is, anysubsequent attempt by one of the replicas to become master of one of thenew groups will fail. If at least one attempt to become master of a newreplica group is successful (shown as the positive exit from 2460), anysubsequent attempt by one of the replicas to become master of theexpanded replica group will be unsuccessful, as in 2480. As previouslynoted, in some embodiments, a replica group may be split into more thantwo new replica groups by a split membership change. In suchembodiments, the techniques described herein may be applied to expandthe original replica group to a sufficient number of replicas topopulate the new replica groups, and then to split the original replicagroup into more than two new replica groups, each of which may thenattempt to elect its own master replica. In such embodiments, theoverlapping quorum requirements described herein may be applicable toall of the post-split replica groups.

One embodiment of a method for releasing mastership of a partition whenit is split is illustrated by the flow diagram in FIG. 25. Asillustrated at 2510, in this example, the method may include the masterfor the expanded replica group shipping a membership change log recordto the other members of the expanded replica group indicating that thegroup is to split into two new replica groups. As in the previousexample, the membership change log record may include informationindicating which replicas are to be included in each of the new replicagroups, a new partition identifier for each of the new replica groups,and a new data range for each of the new replica groups. As illustratedat 2520, the master may continue to hold the lock for the expanded groupuntil it has appended the split log record to its log. After appendingthe split log record, the master replica may give up the mastership ofthe expanded replica group, and then release the external lock for thereplica group (or the partition that it manages), as in 2530.

As illustrated in this example, a replica in the expanded replica groupmay attempt to become master of the expanded replica group, as in 2540(e.g., if it has not received, applied, or committed the split logrecord). Attempting to become master of the expanded replica group mayinclude gathering state information about other replicas in the expandedreplica group and determining whether they support the mastershipattempt, as with any other failover operation. If this attempt to assumemastership is not successful (shown as the negative exit from 2550), themethod may include the replica attempting to become master synchronizingto the split log record, abandoning its attempt to assume membership ofthe expanded replica group, and recognizing its membership in one of thenew replica groups, as in 2570. For example, the attempt to assumemastership of the expanded replica group may fail if the split logrecord is discovered while attempting to fill the failover quorum. Ifthe attempt to assume mastership of the expanded replica group issuccessful (shown as the positive exit from 2550), an attempt by one ofthe replicas to become master of one of the new replica groups will beunsuccessful, as in 2560. In this case, the split operation will failand the split log record will be snipped away from any replica thatappended it to its log stream. As previously noted, the techniquesdescribed herein for splitting a replica group may depend on theoverlapping quorum requirements described above.

In some embodiments, the storage systems described herein may implementfault-tolerant log-handling mechanisms by which log conflicts (e.g., logconflicts caused when logs diverge due to failovers) are detected andresolved. These mechanisms may rely on the safe replication commitstrategy described herein, i.e. that a master may only commit logrecords that it produced during its current master reign. A commit of areplication log record (using the replication mechanisms describedherein) may be defined by the log record achieving minimum durabilitywhen no other log record exists that has the same epoch and sequence,but higher credentials (since the lock generation value indicates thetotal ordering of external lock acquisitions through time, with a highervalue indicating a later lock generation). In other words, this may bethe point at which the log record is intrinsically committed in thesystem (i.e. it is guaranteed to survive), and the only way that amaster can be absolutely sure that a log record is intrinsicallycommitted is if it produced the log record during its current masterreign.

Given this understanding, log conflict detection and resolution may insome embodiments be achieved based on the performance of a comparisonoperation that maps up to four characteristics of log records at thetime that a log record is received (to be appended to the log) to theactions to be taken in response to receiving the log record. Forexample, the comparison operation may examine one or more of:

-   -   the relative difference between log sequence number of the        received log record and the most recently appended log record,        i.e. whether the log sequence number of the received log record        is less than the log sequence number of the most recently        appended log record, equal to the log sequence number of the        most recently appended log record, greater than the log sequence        number of the most recently appended log record by one, or        greater than the log sequence number of the most recently        appended log record by more than one    -   the relative difference between the master epoch of the received        log record and the master epoch of the most recently appended        log record, i.e. whether the master epoch of the received log        record is less than the master epoch of the most recently        appended log record, equal to the master epoch of the most        recently appended log record, greater than the master epoch of        the most recently appended log record by one, or greater than        the master epoch of the most recently appended log record by        more than one    -   the relative difference between the lock generation value of the        received log record and the lock generation value of the most        recently appended log record, i.e. whether the lock generation        value of the received log record is less than the lock        generation value of the most recently appended log record, equal        to the lock generation value of the most recently appended log        record, or greater than the lock generation value of the most        recently appended log record    -   whether or not the incoming log is an “epoch change” log or not,        and if so, whether the previous lock generation value matches        the previous sequence log

In some embodiments, this comparison operation may be expressed as afunction of four parameters with an input domain of all permutations ofthe relative values, and a result range of five directives (or actionsthat may be taken in response). In this example, the possible actionsmay include accepting the log record for appending to the log, droppingthe log record as moot (e.g., if it is already past the log record insequence), caching the log record for the future (e.g., saving it sothat it may be evaluated when the log sequence catches up to it),snipping the log (if a log conflict is detected), or returning an errorindication (if an invalid combination of the input parameter values isdetected). For this example, a matrix of all 96 combinations of theseinput parameters has mapped to the appropriate directives (as shownbelow).

Note that in some embodiments, log conflicts (due to invalid logbranches, etc.) may be detected and resolved during failover. Forexample, a new master hopeful, upon fetching the log tail, may catch upthrough the normal append path, which may result in a log snip. In thiscase, the new master hopeful may break out of the catch-up loop, and mayiterate to fetch the log tail again (e.g., in order to retrieve anearlier log record than it thought it needed prior to the snip). Thequorum peers, upon accepting a log tail (e.g., as part of the replicatetail step) may similarly catch up to the tail, and may snip the log asnecessary. They may then report their most recently flushed log record,which may result in iterating the replicate tail operation from themaster beginning at an earlier point in the log stream. Note that insome embodiments, conflicts may be detected and resolved outside of thefailover process. For example, the data storage system may check for andresolve any detected log conflicts as part of processing every appendmessage, whether or not the append message is received and/or processedduring a failover.

As previously noted, in some embodiments, only a current and activemaster may commit log records (LSNs), and in order to commit a logrecord, it must be guaranteed that the log record will always survive.This means that the log record must meet the durability requirement, andmust also have greater precedence (i.e. higher credentials) than anyother log record that exists. As discussed above, the only way areplication group member can be sure that both of these are true is if(a) the replica produced the record and assigned the precedence (i.e.the LSN credentials) itself based on its current master authority, and(b) it recognizes that the record meets the durability requirement whilestill holding that authority (i.e. while it is still acting as master inthe same master reign, such that the epoch and lock values of the LSNare still the latest in existence). Therefore, in some embodiments, alog record may only be committed by the same master that produced thelog record (and that produced the log record within the current masterreign).

In the failover protocol, this constraint means that a discovered tailmay be replicated, but not committed. This may be acceptable, however,since at the point at which the new master is able to commit a new logthat it produced, all log records leading up to that point will be fullyresolved. The failover protocol may ensure that the log tail catch-upprocess correctly resolves the log based on the guarantee that it willfind that committed log, and the committed log will have a higherauthority than any other log in existence prior to the log beingcommitted. While it may be acceptable (in some embodiments) for this tolog to carry data, this commit may be ensured to happen at the time ofthe failover by requiring the epoch change LSN to commit beforeaccepting new writes. In other embodiments, this may not be required.

In some embodiments, a log conflict detection class may be a staticclass for which the instances are configured to assess the LSN stream.In other words, it may be in these instances that log conflicts aredetected, and in which all snip log cases are defined. The logic of theassessment function (e.g., the comparison operation described above) maybe based on the core invariants provided by the failover protocol, andthe external lock service. The logic for assessing the LSNs in a logstream may be relatively straight-forward when examining LSNs that areall part of the one valid log stream. However, the replication systemsdescribed herein may be designed to be fault tolerant. Therefore, all ofthe cases in which failovers occur that do not include some member ormembers of the one valid log stream may have to be reconciled with themember(s) view of the world once they return. One strategy to deal withthis is to immediately remove any nodes that do not participate in afailover from the replica group, and require them to re-join when theycome back. As noted above, this strategy may put the system at a higherrisk of multiple failures, causing the system to lose quorum, especiallyif the system includes a very large number of small replication groups(e.g., groups that consist of three replicas each most all of the time,as in many of the examples described herein). In addition, as discussedin more detail below, it may take a lot of time and effort to achieveall of the failovers and/or re-joins that may be required following awidespread outage if the system includes a very large number of smallreplication groups. In embodiments that do not remove members fromreplica groups for not participating in a master election, the systemmay be forced to deal with log branching.

In one embodiment, the log stream assessment mechanism may work asfollows. A received LSN may be compared to the current LSN (i.e. themost recently appended LSN), and several comparison values may beproduced (e.g., one for each of the LSN components: epoch, sequencenumber, and lock generation id). These comparison values may beconsidered enums (although they may or may not be represented as enumsin the comparison code). In addition to being dependent on these threecomparison values, the output of the assessment function may in somecases be dependent on a flag that marks a new epoch change LSN.

As noted above, the comparison value for the sequence comparison mayindicate that the received LSN sequence value is one greater than thecurrent LSN sequence value (resulting in a comparison value ofPLUSPLUS), that the received LSN sequence value is more than one greaterthan the current LSN sequence value (resulting in a comparison value ofGTPP), that the received LSN sequence value is the same as the currentLSN sequence value (resulting in a comparison value of EQUAL), or thatthe received LSN sequence value is less than the current LSN sequencevalue (resulting in a comparison value of LESS). Similarly, thecomparison value for the epoch value comparison may indicate that thereceived LSN epoch value is one greater than the current LSN epoch value(resulting in a comparison value of PLUSPLUS), that the received LSNepoch value is more than one greater than the current LSN epoch value(resulting in a comparison value of GTPP), that the received LSN epochvalue is the same as the current LSN epoch value (resulting in acomparison value of EQUAL), or that the received LSN epoch value is lessthan the current LSN epoch value (resulting in a comparison value ofLESS).

In some embodiments, the comparison value for the lock comparison mayhave fewer possible values. In this example, the comparison value forthe lock comparison may indicate that the received LSN lock value isgreater than the current LSN lock value (resulting in a comparison valueof GREATER), that the received LSN lock value is the same as the currentLSN lock value (resulting in a comparison value of EQUAL), or that thereceived LSN lock value is less than the current LSN lock value(resulting in a comparison value of LESS). In some embodiments, thevalue of the new epoch flag may indicate that the received LSN is markedas a “new epoch” LSN (in which case the flag may be set, or “true”), ormay indicate that the received LSN is not marked as a “new epoch” LSN(in which case the flag may be clear, or “false”).

In this example, all assessments are made based on these four pieces ofdata, and all 96 combinations of these values may result in a definitiveaction to be taken in response to receiving a log record. In thisexample, the possible actions that may be taken are:

-   -   ADVANCE: append the received LSN—it is the next LSN in the LSN        stream    -   EPOCH: append the received LSN—it is the next LSN in the LSN        stream (and it also changes the epoch)    -   DROP: drop the received LSN—it is in the past (it has already        been processed or has been superseded)    -   OOC: cache the received LSN—it is in the future (i.e. it was        received out of context), and may be needed later    -   SNIP: snip the log (and then re-assess the received log        record)—the comparison indicates that the log sequence was on an        invalid branch    -   ASSERT: return an error—the combination of parameter values of        the received LSN are invalid (e.g., the failover protocol has        been broken)

The assessment function (and/or the comparison operations thereof) maydepend on the replication protocol and external lock service guaranteesprovided in the system, as expressed in the lemmas and theorems outlinedbelow (for this example). The sequence terminology described earlier maybe assumed in these lemmas and theorems.

The following lemmas may be valid for the example log detection andresolution function described herein:

-   -   Lemma 1: In a given stream branch, the epoch and lock always        increase together. That is, epoch(LSN₁)>epoch(LSN₂) if and only        if lock(LSN₁)>lock(LSN₂) and also lock(LSN₁)>lock(LSN₂) if and        only if epoch(LSN₁)>epoch(LSN₂). Further, if        lock(LSN₁)=lock(LSN₂) then epoch(LSN₁)=epoch(LSN₂), and vice        versa. This is given by the failover protocol.    -   Lemma 2: In order to have an LSN of epoch E, all valid LSN's of        epoch E-1 must first be verified as durable. This is given by        the failover protocol.    -   Lemma 3: Committed LSNs are always included in the valid log        stream during failover. This is given by the failover protocol.    -   Lemma 4: In order to have an LSN of epoch E that is not an epoch        change LSN, the epoch change LSN for epoch E must first be        committed. This is given by the failover protocol.    -   Lemma 5: A unique lock generation is used for each master reign.        This is given by the failover protocol.    -   Lemma 6: Only one lock generation may be active at one time.        This guarantee may be provided by the external lock service.    -   Lemma 7: For an external Lock L, if there are lock acquisition        times of T1, T2, and T1<T2, then lock generation        Lock(T1)<Lock(T2). This guarantee may be provided by the        external lock service.

The following theorems have been proven for the example log detectionand resolution function described herein, but are not shown:

-   -   Theorem 1: If LSN₁ and LSN₂ reside in two different stream        branches, and epoch(LSN₁)>epoch(LSN₂), then lock(LSN₁)        !=lock(LSN₂).    -   Theorem 2: If LSNs with different locks are created in a stream        branch without any other branch creating an LSN in-between, all        LSNs of the first of the two locks must be committed.    -   Theorem 3: If LSN₁ and LSN₂ reside in two different stream        branches, and epoch(LSN₁)>epoch(LSN₂), then        lock(LSN₁)>lock(LSN₂).    -   Theorem 4: If epoch(LSN₁)=epoch(LSN₂), and        lock(LSN₁)<lock(LSN₂), then sequence(LSN₁)<=sequence(LSN₂).    -   Theorem 5: If epoch(LSN₁)=epoch(LSN₂), and lock(LSN₁)        !=lock(LSN₂), and sequence(LSN₁)=sequence(LSN₂), then LSN₂ is an        epoch change LSN.    -   Theorem 6: If sequence(LSN₁)<sequence(LSN₂),        epoch(LSN₁)=epoch(LSN₂), and LSN₂ is marked as an epoch change        LSN, then lock(LSN₁) !=lock(LSN₂).    -   Theorem 7: if lock(LSN₁)>lock(LSN₂), and LSN₂ is not an epoch        change LSN, then epoch(LSN₁) !=epoch(LSN₂).

The example log conflict detection function may be illustrated by thefollowing pseudo code, according to one embodiment.

public class LogConflictDetection { // Possible actions dictated by anassessment // enum Action { DROP, // duplicate delivery, etc. ADVANCE,// normal steady state advancement case - only one case results in thisaction EPOCH, // epoch change advancement case - only one case resultsin this action OOC, // future log - “out of context”. The stream leadingup to this LSN must be filled in SNIP, // log conflict detected (andreplica is on the invalid branch) - log should be snipped ASSERT // acase that should be impossible - therefore, assert error } //Comparisons are: received LSN is _(——) when compared to current LSN(most recently // submitted LSN) // // Valid values for sequence : LESS,EQUAL, PLUSPLUS, GTPP // Valid values for epoch : LESS, EQUAL, PLUSPLUS,GTPP // Valid values for lock : LESS, EQUAL, GREATER // Valid values fornewEpoch : EPOCH, FALSE // static int GREATER = 0; // > - strictlygreater static int GTPP = 0; // >> - more than one greater (greater thanplusplus) static int EQUAL = 1; // == - exactly the same static int LESS= 2; // < - strictly less static int PLUSPLUS = 3; // ++ - one greaterstatic int EPOCH = 0; // newEpoch static int FALSE = 1; // !newEpochstatic Action[ ][ ][ ][ ] matrix = new Action[4][4][3][2]; // currentLSN −> receivedLSN // static  // seq epoch lock newEpoch {matrix[PLUSPLUS][EQUAL][EQUAL][EPOCH] = Action.ASSERT; // ++ == == Tmatrix[PLUSPLUS][EQUAL][EQUAL][FALSE] = Action.ADVANCE; // ++ == == Fmatrix[PLUSPLUS][EQUAL][GREATER][EPOCH] = Action.SNIP; // ++ == > Tmatrix[PLUSPLUS][EQUAL][GREATER][FALSE] = Action.SNIP; // ++ == > Fmatrix[PLUSPLUS][EQUAL][LESS][EPOCH] = Action.ASSERT; // ++ == < Tmatrix[PLUSPLUS][EQUAL][LESS][FALSE] = Action.ASSERT; // ++ == < Fmatrix[PLUSPLUS][PLUSPLUS][EQUAL][EPOCH] = Action.ASSERT; // ++ ++ == Tmatrix[PLUSPLUS][PLUSPLUS][EQUAL][FALSE] = Action.ASSERT; // ++ ++ == Fmatrix[PLUSPLUS][PLUSPLUS][GREATER][EPOCH] = Action.EPOCH; // ++ ++ > Tmatrix[PLUSPLUS][PLUSPLUS][GREATER][FALSE] = Action.SNIP; // ++ ++ > Fmatrix[PLUSPLUS][PLUSPLUS][LESS][EPOCH] = Action.ASSERT; // ++ ++ < Tmatrix[PLUSPLUS][PLUSPLUS][LESS][FALSE] = Action.ASSERT; // ++ ++ < Fmatrix[PLUSPLUS][GTPP][EQUAL][EPOCH] = Action.ASSERT; // ++ >> == Tmatrix[PLUSPLUS][GTPP][EQUAL][FALSE] = Action.ASSERT; // ++ >> == Fmatrix[PLUSPLUS][GTPP][GREATER][EPOCH] = Action.SNIP; // ++ >> > Tmatrix[PLUSPLUS][GTPP][GREATER][FALSE] = Action.SNIP; // ++ >> > Fmatrix[PLUSPLUS][GTPP][LESS][EPOCH] = Action.ASSERT; // ++ >> < Tmatrix[PLUSPLUS][GTPP][LESS][FALSE] = Action.ASSERT; // ++ >> < Fmatrix[PLUSPLUS][LESS][EQUAL][EPOCH] = Action.ASSERT; // ++ < == Tmatrix[PLUSPLUS][LESS][EQUAL][FALSE] = Action.ASSERT; // ++ < == Fmatrix[PLUSPLUS][LESS][GREATER][EPOCH] = Action.ASSERT; // ++ < > Tmatrix[PLUSPLUS][LESS][GREATER][FALSE] = Action.ASSERT; // ++ < > Fmatrix[PLUSPLUS][LESS][LESS][EPOCH] = Action.DROP; // ++ < < Tmatrix[PLUSPLUS][LESS][LESS][FALSE] = Action.DROP; // ++ < < Fmatrix[EQUAL][EQUAL][EQUAL][EPOCH] = Action.DROP; // == == == Tmatrix[EQUAL][EQUAL][EQUAL][FALSE] = Action.DROP; // == == == Fmatrix[EQUAL][EQUAL][GREATER][EPOCH] = Action.SNIP; // == == > Tmatrix[EQUAL][EQUAL][GREATER][FALSE] = Action.ASSERT; // == == > Fmatrix[EQUAL][EQUAL][LESS][EPOCH] = Action.DROP; // == == < Tmatrix[EQUAL][EQUAL][LESS][FALSE] = Action.ASSERT; // == == < Fmatrix[EQUAL][PLUSPLUS][EQUAL][EPOCH] = Action.ASSERT; // == ++ == Tmatrix[EQUAL][PLUSPLUS][EQUAL][FALSE] = Action.ASSERT; // == ++ == Fmatrix[EQUAL][PLUSPLUS][GREATER][EPOCH] = Action.SNIP; // == ++ > Tmatrix[EQUAL][PLUSPLUS][GREATER][FALSE] = Action.SNIP; // == ++ > Fmatrix[EQUAL][PLUSPLUS][LESS][EPOCH] = Action.ASSERT; // == ++ < Tmatrix[EQUAL][PLUSPLUS][LESS][FALSE] = Action.ASSERT; // == ++ < Fmatrix[EQUAL][GTPP][EQUAL][EPOCH] = Action.ASSERT; // == >> == Tmatrix[EQUAL][GTPP][EQUAL][FALSE] = Action.ASSERT; // == >> == Fmatrix[EQUAL][GTPP][GREATER][EPOCH] = Action.SNIP; // == >> > Tmatrix[EQUAL][GTPP][GREATER][FALSE] = Action.SNIP; // == >> > Fmatrix[EQUAL][GTPP][LESS][EPOCH] = Action.ASSERT; // == >> < Tmatrix[EQUAL][GTPP][LESS][FALSE] = Action.ASSERT; // == >> < Fmatrix[EQUAL][LESS][EQUAL][EPOCH] = Action.ASSERT; // == < == Tmatrix[EQUAL][LESS][EQUAL][FALSE] = Action.ASSERT; // == < == Fmatrix[EQUAL][LESS][GREATER][EPOCH] = Action.ASSERT; // == < > Tmatrix[EQUAL][LESS][GREATER][FALSE] = Action.ASSERT; // == < > Fmatrix[EQUAL][LESS][LESS][EPOCH] = Action.DROP; // == < < Tmatrix[EQUAL][LESS][LESS][FALSE] = Action.DROP; // == < < Fmatrix[GTPP][EQUAL][EQUAL][EPOCH] = Action.ASSERT; // >> == == Tmatrix[GTPP][EQUAL][EQUAL][FALSE] = Action.OOC; // >> == == Fmatrix[GTPP][EQUAL][GREATER][EPOCH] = Action.SNIP; // >> == > Tmatrix[GTPP][EQUAL][GREATER][FALSE] = Action.SNIP; // >> == > Fmatrix[GTPP][EQUAL][LESS][EPOCH] = Action.ASSERT; // >> == < Tmatrix[GTPP][EQUAL][LESS][FALSE] = Action.ASSERT; // >> == < Fmatrix[GTPP][PLUSPLUS][EQUAL][EPOCH] = Action.ASSERT; // >> ++ == Tmatrix[GTPP][PLUSPLUS][EQUAL][FALSE] = Action.ASSERT; // >> ++ == Fmatrix[GTPP][PLUSPLUS][GREATER][EPOCH] = Action.OOC; // >> ++ > Tmatrix[GTPP][PLUSPLUS][GREATER][FALSE] = Action.OOC; // >> ++ > Fmatrix[GTPP][PLUSPLUS][LESS][EPOCH] = Action.ASSERT; // >> ++ < Tmatrix[GTPP][PLUSPLUS][LESS][FALSE] = Action.ASSERT; // >> ++ < Fmatrix[GTPP][GTPP][EQUAL][EPOCH] = Action.ASSERT; // >> >> == Tmatrix[GTPP][GTPP][EQUAL][FALSE] = Action.ASSERT; // >> >> == Fmatrix[GTPP][GTPP][GREATER][EPOCH] = Action.OOC; // >> >> > Tmatrix[GTPP][GTPP][GREATER][FALSE] = Action.OOC; // >> >> > Fmatrix[GTPP][GTPP][LESS][EPOCH] = Action.ASSERT; // >> >> < Tmatrix[GTPP][GTPP][LESS][FALSE] = Action.ASSERT; // >> >> < Fmatrix[GTPP][LESS][EQUAL][EPOCH] = Action.ASSERT; // >> < == Tmatrix[GTPP][LESS][EQUAL][FALSE] = Action.ASSERT; // >> < == Fmatrix[GTPP][LESS][GREATER][EPOCH] = Action.ASSERT; // >> < > Tmatrix[GTPP][LESS][GREATER][FALSE] = Action.ASSERT; // >> < > Fmatrix[GTPP][LESS][LESS][EPOCH] = Action.DROP; // >> < < Tmatrix[GTPP][LESS][LESS][FALSE] = ActionDROP; // >> < < Fmatrix[LESS][EQUAL][EQUAL][EPOCH] = Action.DROP; // < == == Tmatrix[LESS][EQUAL][EQUAL][FALSE] = Action.DROP; // < == == Fmatrix[LESS][EQUAL][GREATER][EPOCH] = Action.ASSERT; // < == > Tmatrix[LESS][EQUAL][GREATER][FALSE] = Action.ASSERT; // < == > Fmatrix[LESS][EQUAL][LESS][EPOCH] = Action.DROP; // < == < Tmatrix[LESS][EQUAL][LESS][FALSE] = Action.ASSERT; // < == < Fmatrix[LESS][PLUSPLUS][EQUAL][EPOCH] = Action.ASSERT; // < ++ == Tmatrix[LESS][PLUSPLUS][EQUAL][FALSE] = Action.ASSERT; // < ++ == Fmatrix[LESS][PLUSPLUS][GREATER][EPOCH] = Action.SNIP; // < ++ >Tmatrix[LESS][PLUSPLUS][GREATER][FALSE] = Action.SNIP; // < ++ > Fmatrix[LESS][PLUSPLUS][LESS][EPOCH] = Action.ASSERT; // < ++ < Tmatrix[LESS][PLUSPLUS][LESS][FALSE] = Action.ASSERT; // < ++ < Fmatrix[LESS][GTPP][EQUAL][EPOCH] = Action.ASSERT; // < >> == Tmatrix[LESS][GTPP][EQUAL][FALSE] = Action.ASSERT; // < >> == Fmatrix[LESS][GTPP][GREATER][EPOCH] = Action.SNIP; // < >> > Tmatrix[LESS][GTPP][GREATER][FALSE] = Action.SNIP; // < >> > Fmatrix[LESS][GTPP][LESS][EPOCH] = Action.ASSERT; // < >> < Tmatrix[LESS][GTPP][LESS][FALSE] = Action.ASSERT; // < >> < Fmatrix[LESS][LESS][EQUAL][EPOCH] = Action.ASSERT; // < < == Tmatrix[LESS][LESS][EQUAL][FALSE] = Action.ASSERT; // < < == Fmatrix[LESS][LESS][GREATER][EPOCH] = Action.ASSERT; // < < > Tmatrix[LESS][LESS][GREATER][FALSE] = Action.ASSERT; // < < > Fmatrix[LESS][LESS][LESS][EPOCH] = Action.DROP; // < < < Tmatrix[LESS][LESS][LESS][FALSE] = Action.DROP; // < < < F } /**  *Return the necessary action to perform based on the most recentlysubmitted (appended)  * LSN, the newly received LSN (for append),  * andwhether the received LSN is marked as an epoch change LSN.  */ publicstatic Action assess(LSN currentLSN, LogEntry incomingLog, booleannewEpoch) { LSN receivedLSN = incomingLog.getLSN( ); int sequenceCompare= receivedLSN.seq == currentLSN.seq + 1 ? PLUSPLUS : receivedLSN.seq >currentLSN.seq + 1 ? GTPP : receivedLSN.seq == currentLSN.seq ? EQUAL :LESS; int epochCompare = receivedLSN.epoch == currentLSN.epoch ? EQUAL :receivedLSN.epoch == currentLSN.epoch + 1 ? PLUSPLUS :receivedLSN.epoch > currentLSN.epoch + 1 ? GTPP : LESS; int lockCompare= receivedLSN.lock == currentLSN.lock ? EQUAL : receivedLSN.lock >currentLSN.lock ? GREATER : LESS; int newEpochCompare = newEpoch ? EPOCH: FALSE; Action verdict = matrix[sequenceCompare][epochCompare][lockCompare][newEpochCompare]; //Epoch change is ambiguous with parallel branch transitions if lsn.epochand lsn.seq line  // up (see AMBIGUOUS_EPOCH_CHANGE) // So, verify thetransition is correct by inspecting and comparing the previous lock withthat  // of the current LSN. If not  // a match, then it is the parallelbranch scenario, and is actually a snip case if(verdict == Action.EPOCH){ if(incomingLog.getPriorLock( ) != currentLSN.lock) verdict =Action.SNIP; } return verdict; } }

One embodiment of a method for detecting and resolving log conflicts ina data storage system is illustrated by the flow diagram in FIG. 26. Asillustrated at 2610, in this example, the method may include a slavereplica in a replica group receiving a log record to be appended to itslog stream and metadata related to the log record. The method mayinclude the slave replica comparing the metadata related to the receivedlog record with metadata related to one or more previously appended logrecords, as in 2615. If the comparison indicates that the log stream ofthe slave replica should be snipped (shown as the positive exit from2620), e.g., if a conflict is detected in the compared data thatindicates the log stream of the slave replica is on an invalid branch,the method may include snipping the log stream of the slave replica atthe point of the detected conflict (as in 2625), and then beginning thecomparison operation again (shown as the feedback from 2625 back to2615). Note that in this case, the log stream of the slave replica maybe caught up with the valid stream later (e.g., during a subsequentfailover operation), in some embodiments.

If the comparison indicates that the received log record is the next logrecord that should be appended to the log stream (shown as the positiveexit from 2630), the method may include appending the received logrecord to the log stream, as in 2635. For example, one of thecomparisons that takes place may determine whether the log sequencenumber is next in the log sequence with respect to the log sequencenumbers of the log records already appended in the log (e.g., if it isgreater than the log sequence number of the most recently appended logrecord by one). If so, in some cases, the log record may be appended tothe log stream. In some embodiments, if a flag indicating whether thelog record is associated with an epoch change is set (or true), the logrecord appended to the log stream may indicate the epoch change.

If the comparison does not indicate that the received log record shouldbe appended to the log stream (shown as the negative exit from 2630),but the comparison indicates that the received log record should bedropped, shown as the positive exit from 2640, the method may includedropping the log record, as in 2645. For example, one of the comparisonsthat takes place may determine whether the log record is a duplicate logrecord or a log record that has been superseded. If the comparison doesnot indicate that the received log record should be dropped, but thecomparison indicates that the received log record should be cached as apotential future addition to the log stream (shown as the positive exitfrom 2650), the method may include caching the received log record as apotential future addition to the log, as in 2655. For example, one ofthe comparisons that takes place may determine whether the log sequencenumber of the received log record is not next in sequence, such as ifthe difference between the log sequence number of the received logrecord and the log sequence number of the most recently appended logrecord is greater than one. If so, in some cases, the log record may becached for potential future use. In some embodiments, a cached logrecord may be applied later (e.g., in sequence order), unless adifferent branch is taken, the replica is dropped from group, or anothersituation or state change prevents it being subsequently appended to thelog. Note that if none of the comparisons described above are true(shown as the negative exits from 2620, 2630, 2640, and 2650), thesystem may be in an invalid state, as in 2660, and an error may bereturned. This may correspond to the situation in which the “ASSERT”action is taken in the example pseudo code above (e.g., in response to acombination of metadata comparison results that should not be possiblein the data storage system).

In some embodiments, log branching (and/or the need for varioustechniques to detect and resolve the resulting log conflicts) may beavoided through post-failover rejoins. In some such embodiments, logbranching may be avoided entirely by removing members of a replica groupthat do not participate in a failover election from the replica group,and then replacing them with new members. In some such embodiments, thereplicas that were removed may rejoin using the same mechanisms withwhich any new replica joins a replica group. This alternative approachmay be much simpler to implement than the approach described above, butmay add risk. For example, it may be considered too risky to implementthis alternate approach in a system like that runs 10s to 100s ofthousands (or even millions) of replication groups due to the time andexpense of performing multiple re-join operations for many differentfailovers. For example, the time and effort required for the removedreplica group members to rejoin the replica group may affect thedurability model of the system, and/or the availability of the systemand/or the data maintained thereby. In the case of a large scale event,such as a data center outage, the system would have to work throughmany, many failovers at the same time and process a very large number ofre-join operations to return the system to a stable state. However, itmay be reasonable to implement this alternate approach for singlereplicated databases or in other replicated storage systems on arelatively small scale.

Another embodiment of a method for filling out a failover quorum isillustrated by the flow diagram in FIG. 27. As illustrated in thisexample, the method may include a replica in a replica group initiatingan attempt to become the master for the replica group, as in 2710. Themethod may include the replica that is attempting to become the masterreplica acquiring the external lock associated with the replica groupand/or with the data partition it manages, as in 2720. As describedabove, the method may include the replica that is attempting to becomethe master replica gathering state information from another replica inthe replica group, as in 2730.

As illustrated in this example, if the other replica supports thismastership attempt (shown as the positive exit from 2740), the methodmay include adding the other replica to the failover quorum, as in 2750.On the other hand, if the other replica does not support this mastershipattempt, the other replica is not added to the failover quorum, but maybe retained in the replica group and may be included in a subsequentattempt to reach a failover quorum (whether or not the current attemptis successful). As illustrated in this example, the replica attemptingto become the master replica may continue gathering state informationfrom other replicas in the replica group until the failover quorum isreached (or until a timeout period expires, or until it is clear thatthe failover quorum cannot be reached, in different embodiments). Thisis illustrated in FIG. 27 by the feedback from 2740 to 2730.

In the example illustrated in FIG. 27, once the failover quorum is met,shown as the positive exit from 2760, any replicas that did not supportthe attempt to become master of the replica group may be removed fromthe replica group (as in 2765), and the failover process may continue.In this example, the method may include the replica that is attemptingto become the master replica for the replica group fetching any missingtail of the log stream that is found within one of the other replicas inthe quorum (now the modified replica group), as in 2770. If such a tailis found, the method may include the replica that is attempting tobecome the master replica catching up to the tail and replicating it(e.g., sending its log records to the other replicas in the group in oneor more “append” messages). The replica attempting to become the mastermay then write the first log record of a new epoch (an epoch in which itis the master replica), as in 2780. When this log record is durable, thereplica attempting to become the master replica may commit thenow-replicated tail. As illustrated in this example, the replica that isattempting to become the master may at that point assume mastership forthe modified replica group (e.g., the replica group from which one ormore replicas may have been dropped), as in 2790. Note that, in someembodiments, by dropping any replicas that are not included in thefailover quorum from the replica group, invalid branches in the logstream may be avoided, and the techniques described herein for detectingand resolving such conflicts in the log stream may not be necessary.Also note that in some embodiments, a replica that is dropped from thereplica group may rejoin the replica group at a later time. Rejoiningthe replica group may include discarding the state of the droppedreplica and then synchronizing the replica to the replicas in the quorumfrom scratch (as with any operation to add a new replica to a replicagroup).

As described herein, in some embodiments, a data storage system mayemploy a master/slave replication system with fault tolerance based on aquorum scheme in which write quorums overlap with failoversynchronization quorums to ensure that following a change in mastership,all acknowledged writes have been found. In some embodiments, the datastorage system may be designed to utilize an external lock manager orlock service (e.g. a distributed lock manager) to safely manage theglobal view of the system, including a view indicating which replica ofa replication group is allowed to function as master at any given time.In such embodiments, the distributed lock manager may ensure, through alease mechanism, that one, and only one, replica is ever acting as themaster for each replication group at any given time. The master replicaof the replication group may be the only replica that may master newwrite requests made to the system and directed to the replication group,and may also serve consistent read requests being made to the system anddirected to the replication group. This approach may work well insystems in which the distributed lock manager will remain available tothe replicas, or in systems in which it is acceptable for a givenreplication group to be unavailable for short periods of time. Forexample, if the external lock manager becomes unavailable, then allreplication groups may become masterless (and thus unavailable for bothwrites and consistent reads) as soon as their master leases expire(e.g., within a few seconds, in some systems). The alternate approachesdescribed below may allow both writes and consistent reads to continuein the event that an external lock manager or service that is used toestablish mastership becomes unavailable.

A steady state view of the system may be defined as a state during whicha single master replica coordinates writes to the slave replicas in itsreplica group, and acknowledges the writes when (and if) a quorum of theslave replicas report having persisted them. In some embodiments, duringthis steady state view, consistent read operations may only be served bythe master replica, since the master has been the only replica masteringwrites.

In some embodiments, if the master replica fails, the steady state viewmay cease to function, and a view change may be required in order toresume steady state operations. The view change may involve selecting anew master, issuing it new master credentials that compare greater thanany previous master credentials in existence (i.e. credentials that arethe highest credentials known to any replica in the replica group), andsynchronizing the new master with a quorum of the surviving replicasthat overlaps all previous write quorums used by the previous master. Insome embodiments, the view change may also involve ensuring that allprevious writes satisfy a write quorum that utilizes only the survivingreplicas.

In some embodiments, in order to implement such a view change, thesystem may utilize a consensus mechanism that selects a new masterreplica candidate, and assigns it a new credential that is higher (e.g.,that compares greater) than any previous credential in existence. Insome such embodiments, an external lock manager may be utilized for thispurpose. Once the new master candidate and credentials are selected, thenew master candidate may acquire the failover quorum, and maysynchronize with the quorum according to the applicable failoverprotocol (e.g., the failover quorum described herein). In someembodiments, in order to determine when a view change is required, afailure detection mechanism may also be required. In some embodiments,an external lock manager may be utilized for this purpose as well. Forexample, the external lock manager may maintain a lease with the masterreplica in order to maintain that only one replica is acting as masterfor the steady state view at any single point in time.

One embodiment of a method for employing an external service or manager(e.g., a lock service or lock manager) to select a master replica for areplica group is illustrated by the flow diagram in FIG. 28. Asillustrated at 2810, in this example, the method may include one or morereplicas in a replica group expressing interest in assuming the role ofmaster for the replica group to an external service or manager (e.g., alock service or lock manager). The external service or manager mayselect one of the interested replicas to as a candidate to assume therole of master, may grant a lock for the group to the master candidate,and may assign the highest known credentials to the master candidate, asin 2820. For example, the external service or manager may assign theappropriate sequence number, lock generation identifier and/or epochidentifiers to the master candidate to ensure that only one replica hashighest credentials (e.g., by incrementing one or more of these elementsof the master credentials each time it selects a new master candidate).In some embodiments, the external service or manager may associate alock with each replica group that can be held by at most one replica ata time. In other embodiments, another mechanism may be employed by theexternal service or manager to ensure that only one replica is a validmaster at a time. Note that a change of mastership may be initiated dueto the failure of the master replica (or the computing node on which itis hosted), a loss of communication between the master replica and theexternal service or manager, or another reason, in differentembodiments.

As illustrated at 2830, in this example, the method may includedetermining whether the master candidate succeeds in building a quorumthat supports its attempt to become master replica, as described herein.If not, shown as the negative exit from 2830, the method may include themaster candidate again expressing its interest in assuming the role ofmaster replica, as in 2835. The external service or manager may againselect one of the interested replicas to assume the role of master(either the original master candidate or a new master candidate). Inother words, the method may include repeating the operations illustratedin 2820-2835 for various interested replicas until one is successful inbuilding a quorum of replicas that support the attempt to become masterreplica. This is illustrated by the path from the negative exit of 2830to 2835, and from 2835 back to 2820. If the master candidate succeeds inbuilding a quorum that supports its attempt to become master replica,shown as the positive exit from 2830, the master candidate may assumethe role of master for the replica group, and may begin servicing writesand consistent reads directed to the replica group, as in 2840. In someembodiments, write operations may employ a write quorum (as describedherein), and consistent reads may always be serviced by the masterreplica, since it is the only replica in the group known to have aconsistent view of the data maintained by the replica group.

In some embodiments, a lease may be applied primarily to manageconsistent read operations. For example, in systems in which writeoperations use a quorum scheme, the write operations may be made safewithout the need for a lease. Consistent reads may similarly be madesafe without a lease if they required acquiring quorum as a prerequisiteto responding successfully to the consistent read. However, this woulddecrease system performance, since acquiring quorum is an expensiveoperation that would need to be performed in addition to each read. Notethat, in general, write operations cannot avoid the expense of acquiringa quorum because all write operations must be performed on all replicas.In contrast, consistent read operations may be performed only on themaster, e.g., as long as it is certain that there is only one master. Onthe other hand, as described in reference to FIG. 17B, eventuallyconsistent read operations may be served by any replica in theappropriate replica group. In some embodiments, the use of a lease mayensure that there is only one master at all times regardless of networkpartitions, etc., that may cause false positives in the failuredetector.

In some systems that rely on an external lock manager for master leases,the system may also utilize the external lock manager for maintainingthe steady state view for master writes. For example, in order tomaintain a lease, a heart-beat mechanism may be employed between theexternal lock manager and the current master replica, such that thelease is continued or renewed periodically as long as the external lockmanager and the master replica remain in communication with each otherand no other replica assumes the role of master replica for the replicagroup. In some embodiments, if the lease is lost, the mastership forboth consistent read operations and write operations may be given up.While this strategy may have the advantage of simplicity, it may alsohave the disadvantage that if the external lease mechanism fails orotherwise becomes unavailable, the master may be lost, and the steadystate view may cease to function. In some embodiments, since this leasemechanism may be this same mechanism that provides consensus forselecting the next master and its new credentials, it may not bepossible to establish a new steady state view if the external leasemechanism fails or otherwise becomes unavailable. This may lead to aservice outage. The “blast radius” (i.e. the affected range) of such aservice outage may include all master/slave views that were utilizingthe failed lease manager, which may be unacceptable in some systems.

One embodiment of a method for employing a heart-beat mechanism betweenan external service or manager (e.g., a lock service or lock manager)and a master replica for a replica group is illustrated by the flowdiagram in FIG. 29. As illustrated in this example, the method mayinclude a master candidate receiving a lease from an external service ormanager, and assuming the role of master for its replica group, as in2910. The method may also include the master replica servicing writeoperations and consistent read operations that are directed to thereplica group, as in 2920. As illustrated in this example, the methodmay include, after a pre-determined time period, the master replicasending a heart-beat message to the external service/manager, as in2930. In other words, the new master may begin a heart-beat process withthe external service/manager in order to maintain (or periodicallyrenew) the lease. In such embodiments, the heart-beat time period may beless than the lease period.

As illustrated in FIG. 29, if the new master receives a new (or renewed)lease from the external service/manager in response to sending theheart-beat message (shown as the positive exit from 2940), the methodmay include repeating the operations illustrated at 2920-2940 as long asthe new master continues to receive new leases in response to theheart-beat message. If the new master does not receive a new (orrenewed) lease from the external service/manager in response to sendingthe heart-beat message (shown as the negative exit from 2940), themethod may include the master losing its lease, and no longer servicingthe write operations and consistent read operations that are directed tothe replica group, as in 2950. In this case, write operations and/orconsistent read operations may not be serviced until another replicabecomes the master replica for the replica group.

As described herein, in some embodiments, write operations may employ aquorum scheme. In some embodiments, this mechanism may be leveraged toallow write operations to continue to be available following the loss ofany external lease mechanism. For example, rather than giving upmastership for write purposes when the lease may no longer be maintained(due to problems with the lease manager, or simply due to communicationissues with the lease manager), the steady state master may simplycontinue to act in the role of master for the replica group so long asenough replicas only support write quorums for writes mastered by areplica with the greatest credentials that the replica has ever seen.Since all write quorums must intersect all failover quorums, if anotherreplica succeeds in achieving a failover quorum (which may be apre-requisite for the replica to become master), then any previousmaster will no longer be able to achieve write quorums for writes thatit tries to master. Thus, it may be safe for the new master to emergewithout the possibility of having two replicas attempting to act asmasters for write operations directed to the replica group.

In some embodiments, to enable this approach, the failover protocoldescribed above may be extended as follows: When a replica participatesin a new failover quorum, it may first validate that it has never seen ahigher master credential or else may refuse to be part of the quorum. Inother words, a replica cannot support failover quorums for credentialsthat are not greater than any it has previously seen, according to thefailover protocol described above. The replica may durably remember(e.g., persist to disk or write to another type of persistent memory)the new highest master credential, and may not be able to respond as aparticipant in the quorum until the new highest master credential isdurably remembered. At that point, the replica may agree to reject anyattempted write operation that is mastered under a lesser mastercredential.

In some embodiments, in order to establish a new master for a new steadystate view, the system may require a consensus mechanism for determiningthe replica that may be master next, and its new credentials. The newcredentials may be guaranteed to be greater than any previous mastercredentials. Using the protocol described above, once the steady stateview is established, it may be maintained for write operations withoutthe need to maintain (or even know about) any leases.

One embodiment of a method for continuing to service write operationswhen an external service or manager (e.g., a lock service or lockmanager) is unavailable is illustrated by the flow diagram in FIG. 30.As illustrated in this example, the method may include a given replicareceiving credentials from an external service or manager, persistingthose credentials on the local node (e.g., persisting them to disk orwriting them to another type of persistent memory), and assuming therole of master for its replica group, as in 3010. As illustrated at3020, the method may also include the master replica servicing writeoperations that are directed to the replica group, which may includeattempting to obtain a write quorum for each of the write operations (asdescribed in detail herein).

As illustrated in FIG. 30, the master replica may lose communicationwith the external service/manager (as in 3030). For example, theexternal manager/service (or the computing node or nodes on which it ishosted) may fail, communication between the master replica and theexternal service/manager may fail, or the heart-beat (or the responsethereto) may be lost. However, the master may continue servicing writeoperations that are directed to the replica group, which may includeattempting to obtain a write quorum for each of the write operations, asin 3040. If the write quorum is met for a given replicated writeoperation (shown as the positive exit from 3050), the method may includecommitting that write operation in the data store, as in 3055. If thewrite quorum is not met for a given replicated write operation (shown asthe negative exit from 3050), that write operation may not be committedin the data store, as in 3060.

As illustrated in this example, if no other replica has seen highercredentials than those held by the current master (shown as the negativeexit from 3070), the method may include repeating the operationsillustrated as 3040-3070. In other words, until another replica sees (orholds) higher credentials than those that were assigned to the givenreplica at 3010, the given replica may continue to act as master for thereplica group, and may continue to service write operations directed tothe replica group (committing those for which a write quorum isachieved). However, if (at any point) another replica sees (or holds)higher credentials than the current master (i.e. credentials higher thanthose that were assigned to the given replica at 3010), the givenreplica may give up mastership of the replica group and may no longerservice write operations that are directed to the replica group. This isillustrated in FIG. 30 by the negative exit from 3070, and element 3080.

Another embodiment of a method for continuing to service writeoperations when an external service or manager (e.g., a lock service orlock manager) is unavailable is illustrated by the flow diagram in FIG.31. As in the previous example, the method may include a given replicareceiving credentials from an external service or manager, persistingthose credentials on the local node (e.g., persisting them to disk orwriting them to another type of persistent memory), and assuming therole of master for its replica group, as in 3110. The method may alsoinclude the master replica servicing write operations that are directedto the replica group, which may include attempting to obtain a writequorum for each of the write operations, as in 3120. As describedherein, in some embodiments, the master replica may implement aheart-beat mechanism in which messages are exchanged between the masterreplica and the external service/manager in order to maintain themastership of the given replica and to ensure that only one replica actsin the role of master replica for the replica group at a time. In suchembodiments, the heart-beat between the given replica and the externalservice/manager may fail, as in 3130. For example, the externalmanager/service (or the computing node or nodes on which it is hosted)may fail, or communication between the master replica and the externalservice/manager may fail, causing the heart-beat (or the responsethereto) to be lost. However, the master may continue servicing writeoperations that are directed to the replica group (which may host aparticular data partition), which may include attempting to obtain awrite quorum for each of the write operations, as in 3140. As in theprevious example, write operations for which the write quorum isachieved may be committed in the data store (not shown).

As illustrated in this example, if another replica assumes the role ofmaster replica for the replica group or is determined to be attemptingto assume the role of master replica for the replica group (shown as thepositive exit from 3150), the given replica may refrain from servicingany subsequently requested write operations that are directed to thereplica group, as in 3160. For example, if the given replica is asked toparticipate in a quorum for a new master election, or once the givenreplica determines (after the fact) that another replica has assumedmastership of the replica group through a failover operation of which itwas unaware, it may refrain from servicing write operations directed toits replica group. Instead, only the new master (once it has assumed therole of master) may service any subsequent write operations that aredirected to the replica group, as in 3170. However, until anotherreplica assumes (or attempts to assume) the role of master replica forthe replica group, the given replica may continue to service writeoperations that are directed to the replica group, regardless of thestate of the external manager/service. This is illustrated in FIG. 31 bythe feedback from the negative exit of 3150 to 3140.

The approach described above may allow write operations to continue evenwhen an external lock/lease service or manager is unavailable. In someembodiments, the system may require acquiring quorum for consistentreads that will overlap the failover quorums without a lease, as is thecase with write operations. For example, in some embodiments, the quorummay only be required when the external lock/lease service or manager isunavailable. Under these circumstances (i.e. when the externallock/lease manager is unavailable), the performance of the system may bedegraded, since consistent reads may suddenly become much moreexpensive. (i.e. they may be much slower). Such an approach may also addload to the other replicas in the replica group, which may impacteventually consistent read operations, as well.

In some embodiments, another approach may be utilized to allowconsistent read operations to continue when an external lock/leaseservice or manager is unavailable without going into a significantlydegraded mode. For example, the system may utilize a local leasemechanism (i.e. the lease mechanism may be implemented in thereplication group itself) for this purpose. In some embodiments, thesystem may implement a heart-beat mechanism between the current masterreplica and the other replicas in the replica group (i.e. the slavereplicas) that is used to ensure that all replicas have the latestinformation (e.g., that nothing has been missed). In some embodiments,this heart-beat mechanism may be implemented using LSNs (replicatedwrites) issued by the current master, and the heart-beat messages mayalso to be used as the lease mechanism for consistent read operations.Because they are expressed as replicated writes, the heart-beat messagesmay only succeed in obtaining the write quorum (and taking effect) if noother master has emerged (as with any other replicated writes). Theheart-beat messages may include an indication of the lease and/or anindication of a lease period (e.g., a configurable time interval that isgreater than the heart-beat interval), such that they establish a leasefor the specified period if they are committed.

One embodiment of a method for employing a series of local leases todetermine the replica authorized to service consistent read operationsis illustrated by the flow diagram in FIG. 32. As in previous examples,the method may include a given replica receiving credentials from anexternal service or manager, persisting those credentials on the localnode (e.g., persisting them to disk or writing them to another type ofpersistent memory), and assuming the role of master for its replicagroup, as in 3210. As illustrated in FIG. 32, the method may include themaster replica sending a replicated write that indicates a lease to therest of the replica group, and attempting to obtain a write quorum forthat write operation, as in 3220. In this example, the lease mayrepresent the authorization of a replica to act as the master replicafor its replica group for a pre-determined amount of time (the leaseperiod). The lease message may also include an identifier of the replicathat mastered the message and/or any of the other information typicallyincluded in replicated writes in the system. If the write operationindicating the lease achieves a write quorum (shown as the positive exitfrom 3230), the method may include the master replica servicingconsistent read operations that are directed to the replica group (datapartition), and servicing write operations that are directed to thereplica group (data partition), committing them to the data store if awrite quorum is achieved, as in 3240.

As illustrated in this example, the method may include, after apre-determined time period (the heart-beat period), the master replicasending the next lease message to the rest of replica group as areplicated write (as in 3250). In other words, once a given replica hasbeen assigned credentials (and thus, the authority to attempt to becomethe master replica for its replica group) by an external service ormanager (or by other means), and the replica has become the masterreplica for the replica group, that master replica may implement a leaselocally (within the replica group), rather than relying on an externalservice or manager to maintain its authority to service consistent readoperations directed to the replica group (data partition), i.e. to renewthe lease for an additional lease period. For example, in someembodiments, the leases may be originated, maintained and/or renewedusing local heart-beat messages for which the heart-beat period is lessthan the lease period. As illustrated in this example, if the writeoperation indicating the lease (heart-beat message) achieves a writequorum within the lease period (shown as the positive exit from 3260),the method may include repeating the operations illustrated as 3240-3260until a subsequent attempt to renew the lease fails (e.g., until asubsequent write operation indicating a lease fails to achieve the writequorum within the current lease period).

If one of the write operations indicating the lease (e.g., anorigination or renewal of the lease) does not achieve a write quorum(shown as the negative exit from 3230 or the negative exit from 3260),the method may include the master refraining from servicing subsequentconsistent read operations that are directed to the replica group, as in3270. However, the master replica may not refrain from performingsubsequent write operations in response to a failure to achieve a writequorum for a lease. Instead, the master may refrain from performingwrite operations only when (and if) it becomes aware of anotherreplica's attempt to become a new master replica for the group(regardless of whether that attempt has successfully completed). Asillustrated in this example, following a failure to renew a least withinthe current lease period, the master replica may generate a new lease(assuming no other replica has assumed the role of master replica forthe replica group). This is illustrated in FIG. 32 as the path from 3270back to 3220. Note that in some embodiments, if a write quorum for alease renewal is not reached during a current lease period, but isreached shortly afterward (e.g., before another replica has had a chanceto assume the role of master replica for the group or to attempt toassume the role of master replica), the master replica may resumeservicing consistent reads that are directed to the replica groupwithout having to initiate another write operation indicating a newlease or a new lease renewal.

In some embodiments, a local lease mechanism may also be utilized tomaintain the master replica for consistent read operations whenmastership of a replica group changes. One embodiment of a method fordetermining the replica authorized to service consistent read operationswhen mastership of a replica group changes is illustrated by the flowdiagram in FIG. 33. As illustrated in this example, the method mayinclude a given replica receiving credentials from an external serviceor manager, persisting those credentials on the local node (e.g.,persisting them to disk or writing them to another type of persistentmemory), and assuming the role of master for its replica group, as in3310. The method may include the master replica sending a messageindicating a lease (e.g., the origination of a new lease or the renewalof an existing lease) to the rest of the replica group as a replicatedwrite operation, and attempting to obtain a write quorum for that writeoperation, as in 3320. As described herein, the lease may in someembodiments represent the authorization of the replica to act as themaster replica for its replica group for a pre-determined amount of time(the lease period). If the write quorum is achieved (shown as thepositive exit from 3330), the method may include the master replicaservicing consistent read operations that are directed to the replicagroup (or a corresponding data partition for which it stores data), andservicing write operations that are directed to the replica group (datapartition), committing them to the data store if a write quorum isachieved, as in 3340. If the write quorum is not achieved within thelease period (shown as the negative exit from 3330), the given replicamay refrain from servicing subsequent consistent read operations thatare directed to the replica group (data partition), as in 3380. In thiscase, the master replica may still service write operations, which mayuse a quorum mechanism to determine whether they should be committed.

In some embodiments, until another replica assumes the role of masterreplica for the replica group (or determines that another replica isattempting to assume the role of master replica), the current masterreplica may continue to service consistent read operations and/or writeoperations that are directed to the replica group (data partition),regardless of the state of the external service/manager. This isillustrated in FIG. 33 by the feedback from the negative exit of 3350 to3340. As described herein, this may include generating and sendingadditional local lease messages to the other replicas in the replicagroup until and unless another replica assumes (or is determined to beattempting to assume) the role of master replica for the replica group.As illustrated in FIG. 33, if another replica assumes (or is attemptingto assume) the role of master replica for the replica group (shown asthe positive exit from 3350), and the most recent lease generated by thegiven replica has expired (shown as the positive exit from 3360), themethod may include the given replica refraining from servicingsubsequent consistent read operations that are directed to the replicagroup (data partition), as in 3380. On the other hand, if anotherreplica assumes (or is attempting to assume) the role of master replicafor the replica group (shown as the positive exit from 3350), but themost recent lease generated by the given replica has not expired (shownas the negative exit from 3360), the method may include the givenreplica continuing to service consistent read operations that aredirected to the replica group (data partition) until the lease expires,as in 3370. This is illustrated in FIG. 32 by the feedback from thenegative exit of 3360 to 3370, and the path from 3370 back to 3360. Inother words, in some embodiments, a new master replica that has built aquorum may not take over the responsibility of mastering consistent readoperations until an active local lease has expired.

Note that in some embodiments, the master replica may start its leasetimer immediately upon issuing the lease message (e.g., before sendingit out to the other members of the replica group for quorum), and maynot use the lease (i.e. may not master any consistent read operations)until it has received sufficient acknowledgements indicating that thewrite of the lease message has reached quorum (which may represent thepoint at which the write may be committed). Each replica receiving thelease message may independently note the current time (e.g., asindicated by their local clock) when they process the heart-beat (lease)write operation. Any heart-beat (lease) write operation that achievesquorum may be guaranteed to be found by the failover quorum during afailover steady state view change. As noted above, in some embodiments,the new master having achieved the failover quorum may allow any foundlease (i.e. the latest heart-beat processed) to expire prior to takingover the role of master replica for the replica group. At that time, thenew master replica may be certain that any previous master will not beusing that lease, and that any newer lease that was not found did notreach the write quorum (thus, a previous master could not be using iteither). In some embodiments, the approach described above may ensurethat two replicas cannot be acting as master replica for a replica groupat the same time. Note that in some embodiments, if a previous master(i.e. the issuer of a currently active lease) participates in thefailover quorum that establishes a new master, it may not honor thelease it had previously issued (e.g., it may give up or cancel the leaseprior to the expiration of the lease period). In such embodiments, thenew master may not need to wait for the lease period to expire beforeassuming the role of master replica for the replica group.

Another embodiment of a method for determining the replica authorized toservice consistent read operations when mastership of a replica groupchanges is illustrated by the flow diagram in FIG. 34. As illustrated inthis example, the method may include a given replica receivingcredentials from an external service or manager, persisting thosecredentials on the local node (e.g., persisting them to disk or writingthem to another type of persistent memory), and assuming the role ofmaster for its replica group, as in 3410. The method may include themaster replica sending a message indicating a lease (e.g., theorigination of a new lease or the renewal of an existing lease) to therest of the replica group as a replicated write operation, and obtaininga write quorum for that write operation, as in 3420. As describedherein, the lease may in some embodiments represent the authorization ofthe replica to act as the master replica for its replica group for apre-determined amount of time (the lease period). As illustrated in thisexample, the method may include the master replica servicing consistentread operations that are directed to the replica group (or acorresponding data partition for which it stores data), and servicingwrite operations that are directed to the replica group (datapartition), committing them to the data store if a write quorum isachieved, as in 3430. The method may also include another replicaattempting to assume mastership of the replica group, and obtaining aquorum that supports its attempt to become master of the replica group,as in 3440. As illustrated in this example, the method may include theother replica (i.e. the replica that is in the process of assumingmastership) sending a message indicating a second lease (e.g., theorigination of a new lease for the replica that is assuming mastership)to the rest of the replica group as a replicated write operation, andobtaining a write quorum for that message, as in 3450.

As illustrated in FIG. 34, the method may include determining whetherthe previous lease, i.e. the most recent lease generated by the givenreplica (the current master), has expired, as in 3460. If not, shown asthe negative exit from 3460, the method may include the given replicacontinuing to service any subsequent consistent read operations that aredirected to the replica group (data partition), as in 3465, until theprevious lease expires. This is illustrated in FIG. 34 by the feedbackfrom the negative exit of 3460 to 3465, and the path from 3465 back to3460. If the previous lease has expired (or once it subsequentlyexpires), shown as the positive exit from 3460, the method may includethe given replica refraining from servicing any subsequent consistentread operations that are directed to the replica group (data partition),as in 3470, and the other replica (the replica that is assuming the roleof master for the replica group) beginning to service any consistentread operations under the authority of the second lease, as in 3480.

Note that in some embodiments, timing may only be measured by the localclock(s) on each computing node. In such embodiments, the techniquesdescribed herein may not depend on clock skew across servers. Instead,they may only depend on the local clocks of individual servers runningat the same rate. This may also be a base requirement necessary for anexternal lock/lease service or manager to function properly.

In some embodiments, the techniques described herein for managingmastership of write operations and consistent read operations withoutrelying on an external lock/lease service or manager may depend onquorum and persistence facilities already built into the system. In someembodiments, a consensus mechanism may still be required for enacting asteady state view change, and for determining the new mastercredentials. However, in some embodiments, the consensus mechanismsdescribed herein may be replaced with other consensus mechanisms(perhaps within the replica group itself), which may eliminate thedependency on an external lock/lease service or manager entirely.

One computing node that may be suitable for implementation of a datastorage service that employs the techniques described herein isillustrated in FIG. 35. Computing node 3500 may include functionality toprovide any or all of the components of a system that implements such adata storage service, or multiple computing nodes similar to ordifferent from computing node 3500 may collectively provide thisfunctionality, in different embodiments. For example, in variousembodiments, one or more computing nodes 3500 may implement any numberof storage service clients 110, a front end module 140, any number ofauto admin instances 150, any number of storage devices (such as storagenode instances 160), and/or any other components of a Web servicesplatform 130, an auto admin cluster, or external resources that interactwith Web services platform 130 (such as external workflow component 170or external storage service 180). Any number of those storage nodeinstances 160 may each host one or more replicas of various datapartitions and/or metadata associated therewith. For example, any givenstorage node instance 160 may host a replica acting as master replicasfor its replica group and/or a replica acting as a slave replica in itsreplica group. In various embodiments, any or all of the techniquesdescribed herein for partitioning, replication, and/or managementthereof may be performed by one or more components of the storage nodeinstances 160 that host a master replica and/or a slave replica, such aspartition manager 270 and replication and failover component 275illustrated in FIG. 2C. In some embodiments that include multiplecomputing nodes 3500, all of the computing nodes 3500 may include thesame or similar hardware components, software components, andfunctionality, while in other embodiments, the computing nodes 3500comprising a computing system configured to implement the functionalitydescribed herein may include a wide variety of hardware components,software components, and functionality. In some embodiments, multiplecomputing nodes 3500 that collectively implement a data storage servicemay be components of a larger shared resource system or grid computingsystem.

In the illustrated embodiment, computing node 3500 includes one or moreprocessors 3510 coupled to a system memory 3520 via an input/output(I/O) interface 3530. Computing node 3500 further includes a networkinterface 3540 coupled to I/O interface 3530, and one or moreinput/output devices 3550. As noted above, in some embodiments, a givennode may implement the functionality of more than one component of asystem that manages and maintains data in tables (e.g., in anon-relational database) on behalf of data storage service clients, suchas that described herein. In various embodiments, a computing node 3500may be a uniprocessor system including one processor 3510, or amultiprocessor system including several processors 3510 (e.g., two,four, eight, or another suitable number). Processors 3510 may be anysuitable processor capable of executing instructions. For example, invarious embodiments processors 3510 may be general-purpose or embeddedprocessors implementing any of a variety of instruction setarchitectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, orany other suitable ISA. In multiprocessor systems, each of processors3510 may commonly, but not necessarily, implement the same ISA.Similarly, in a distributed computing system such as one thatcollectively implements a data storage service, each of the computingnodes may implement the same ISA, or individual computing nodes and/orreplica groups of nodes may implement different ISAs.

In some embodiments, system memory 3520 may include a non-transitory,computer-readable storage medium configured to store programinstructions and/or data accessible by processor(s) 3510. In variousembodiments, system memory 3520 may be implemented using any suitablememory technology, such as static random access memory (SRAM),synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or anyother type of memory. In the illustrated embodiment, programinstructions and data implementing desired functions, such as thosedescribed above, are shown stored within system memory 3520 as programinstructions 3525 and data storage 3535, respectively. For example,program instruction 3525 may include program instructions that whenexecuted on processor(s) 3510 implement any or all of a storage serviceclient 110, a front end module 140 (which may include a user interface),an auto admin instance 150, a storage node instance 160, an adminconsole 265, a request router, a staging host, one or more metadatatables, an external workflow component 170, an external storage service180, and/or any other components, modules, or sub-modules of a systemthat provides the data storage service described herein. Programinstructions 3525 may also include program instructions configured toimplement additional functionality of a system that implements a datastorage service not described herein.

Data storage 3535 may in various embodiments include collections of datamaintained by a data storage service on behalf of its clients/users,and/or metadata used by a computing system that implements such aservice, as described herein (including, but not limited to, tablesmanaged and maintained on behalf of clients/users of the service,metadata tables, business rules, partition maps, routing tables,indexes, namespaces and/or partitions thereof, service level agreementparameter values, subscriber preferences and/or account information,performance data, and/or resource usage data). In other embodiments,program instructions and/or data as described herein for implementing adata storage service that employs the techniques described above may bereceived, sent or stored upon different types of computer-readable mediaor on similar media separate from system memory 3520 or computing node3500. Generally speaking, a computer-readable medium may include storagemedia or memory media such as magnetic or optical media, e.g., disk orCD/DVD-ROM coupled to computing node 3500 via I/O interface 3530.Program instructions and data stored on a computer-readable storagemedium may be transmitted to a computing node 3500 for execution by aprocessor 3510 a by transmission media or signals such as electrical,electromagnetic, or digital signals, which may be conveyed via acommunication medium such as a network and/or a wireless link, such asmay be implemented via network interface 3540.

In one embodiment, I/O interface 3530 may be configured to coordinateI/O traffic between processor(s) 3510, system memory 3520, and anyperipheral devices in the computing node, including network interface3540 or other peripheral interfaces, such as input/output devices 3550.In some embodiments, I/O interface 3530 may perform any necessaryprotocol, timing or other data transformations to convert data signalsfrom one component (e.g., system memory 3520) into a format suitable foruse by another component (e.g., processor 3510). In some embodiments,I/O interface 3530 may include support for devices attached throughvarious types of peripheral buses, such as a variant of the PeripheralComponent Interconnect (PCI) bus standard or the Universal Serial Bus(USB) standard, for example. In some embodiments, the function of I/Ointerface 3530 may be split into two or more separate components, suchas a north bridge and a south bridge, for example. Also, in someembodiments some or all of the functionality of I/O interface 3530, suchas an interface to system memory 3520, may be incorporated directly intoprocessor 3510.

Network interface 3540 may be configured to allow data to be exchangedbetween computing node 3500 and other devices attached to a network(such as other computer systems, communication devices, input/outputdevices, or external storage devices), or between other nodes in asystem providing shared computing services. In various embodiments,network interface 3540 may support communication via wired or wirelessgeneral data networks, such as any suitable type of Ethernet network,for example; via telecommunications/telephony networks such as analogvoice networks or digital fiber communications networks; via storagearea networks such as Fibre Channel SANs, or via any other suitable typeof network and/or protocol.

Input/output devices 3550 may, in some embodiments, include one or moredisplay terminals, keyboards, keypads, touchpads, scanning devices,voice or optical recognition devices, or any other devices suitable forentering or retrieving data by one or more computing nodes 3500.Multiple input/output devices 3550 may be present in computing node 3500or may be distributed on various computing nodes of a system that isconfigured to implement a data storage service. In some embodiments,similar input/output devices may be separate from computing node 3500and may interact with one or more computing nodes of a system through awired or wireless connection, such as over network interface 3540.

Storage service clients (e.g., users, subscribers and/or clientapplications) may interact with a data storage service such as thatdescribed herein in various ways in different embodiments, such as tosubmit requests for service (including, but not limited to, requests tostore, retrieve and/or update items in tables, or requests torepartition a table), and to receive results. For example, somesubscribers to the service may have physical access to computing node3500, and if so, may interact with various input/output devices 3550 toprovide and/or receive information. Alternatively, other clients/usersmay use client computing systems to access the system, such as remotelyvia network interface 3540 (e.g., via the Internet and/or the World WideWeb). In addition, some or all of the computing nodes of a systemproviding the service may provide various feedback or other generaltypes of information to clients/users (e.g., in response to userrequests) via one or more input/output devices 3550.

Those skilled in the art will appreciate that computing node 3500 ismerely illustrative and is not intended to limit the scope ofembodiments. In particular, the computing system and devices may includeany combination of hardware or software that can perform the indicatedfunctions, including computers, network devices, interne appliances,PDAs, wireless phones, pagers, etc. Computing node 3500 may also beconnected to other devices that are not illustrated, in someembodiments. In addition, the functionality provided by the illustratedcomponents may in some embodiments be combined in fewer components ordistributed in additional components. Similarly, in some embodiments thefunctionality of some of the illustrated components may not be providedand/or other additional functionality may be available.

Those skilled in the art will also appreciate that, while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memoryand other storage devices for purposes of memory management and dataintegrity. Alternatively, in other embodiments some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated computing system via inter-computercommunication. Some or all of the system components or data structuresmay also be stored (e.g., as instructions or structured data) on acomputer-readable storage medium or a portable article to be read by anappropriate drive, various examples of which are described above. Insome embodiments, instructions stored on a computer-readable storagemedium separate from computing node 3500 may be transmitted to computingnode 3500 via transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a network and/or a wireless link. Various embodiments mayfurther include receiving, sending or storing instructions and/or dataimplemented in accordance with the foregoing description upon acomputer-readable storage medium. Accordingly, different embodiments maybe practiced with other computer system configurations.

Note that while several examples described herein are directed to theapplication of various techniques in systems that include anon-relational database, in other embodiments these techniques may beapplied in systems in which the non-relational data store is implementedusing a different storage paradigm.

Those skilled in the art will appreciate that in some embodiments thefunctionality provided by the methods discussed above may be provided inalternative ways, such as being split among more software modules orroutines or consolidated into fewer modules or routines. Similarly, insome embodiments illustrated methods may provide more or lessfunctionality than is described, such as when other illustrated methodsinstead lack or include such functionality respectively, or when theamount of functionality that is provided is altered. In addition, whilevarious operations may be illustrated as being performed in a particularmanner (e.g., in serial or in parallel) and/or in a particular order,those skilled in the art will appreciate that in other embodiments theoperations may be performed in other orders and in other manners. Thoseskilled in the art will also appreciate that the data structuresdiscussed above may be structured in different manners, such as byhaving a single data structure split into multiple data structures or byhaving multiple data structures consolidated into a single datastructure. Similarly, in some embodiments illustrated data structuresmay store more or less information than is described, such as when otherillustrated data structures instead lack or include such informationrespectively, or when the amount or types of information that is storedis altered. The various methods as depicted in the figures and describedherein represent illustrative embodiments of methods. The methods may beimplemented in software, in hardware, or in a combination thereof invarious embodiments. Similarly, the order of any method may be changed,and various elements may be added, reordered, combined, omitted,modified, etc., in various embodiments.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the appended claims and the elements recited therein. Inaddition, while certain aspects are presented below in certain claimforms, the inventors contemplate the various aspects in any availableclaim form. For example, while only some aspects may currently berecited as being embodied in a computer readable storage medium, otheraspects may likewise be so embodied. Various modifications and changesmay be made as would be obvious to a person skilled in the art havingthe benefit of this disclosure. It is intended to embrace all suchmodifications and changes and, accordingly, the above description to beregarded in an illustrative rather than a restrictive sense.

1. A system, comprising: a plurality of computing nodes, each comprisingat least one processor and memory, wherein the plurality of computingnodes is configured to implement a data storage service; wherein thedata storage service maintains data on behalf of one or more storageservice clients, wherein maintaining the data comprises storing two ormore replicas of the data on respective computing nodes in the system,wherein the two or more replicas make up a replica group, wherein thereplica group maintains an indicator of its membership version, whereinthe membership version is updated each time the membership of thereplica group changes, wherein at most one of the replicas in thereplica group can act as a master replica for the replica group at atime, and wherein replicas in the replica group that are not acting asthe master replica act as slave replicas in the replica group; whereinone of the two or more replicas is configured to attempt to assume arole of master replica for the replica group, and wherein attempting toassume the role of master replica comprises: acquiring a lock associatedwith the replica group, wherein only one of the two or more replicas canhold the lock at a time; gathering state information from at least someof the two or more replicas other than the one of the replicas; for eachof the at least some of the two or more replicas: determining, dependenton the state information, whether the replica supports the attempt toassume the role of master replica; and in response to determining thatthe replica supports the attempt to assume the role of master replica,including the replica in a failover quorum; and wherein the one of thereplicas is further configured to: determine whether at least apre-determined number of replicas are included in the failover quorum;and assume the role of master replica for the two or more replicas inresponse to determining that at least the pre-determined number ofreplicas are included in the failover quorum.
 2. The system of claim 1,wherein determining whether a replica supports an attempt to assume therole of master replica comprises one or more of: determining whether thereplica has observed a more recent membership version for the replicasstoring the data than a most recent membership version observed by theone of the replicas attempting to assume the role of master replica,wherein the membership version is incremented each time a membershipchange is made in the replica group; determining whether the replica isno longer hosted on the computing node from which the state informationfor the replica is gathered; or determining whether the replica has seena more recent value for the lock than a most recent lock value observedby the one of the replicas attempting to assume the role of masterreplica, wherein the lock value is incremented each time the lock isacquired by a different replica.
 3. The system of claim 1, wherein theone of the replicas is further configured to abandon the attempt toassume the role of master replica for the replica group in response todetermining that fewer than the pre-determined number of replicas areincluded in the failover quorum.
 4. The system of claim 1, wherein theone of the replicas is further configured to, prior to assuming the roleof master replica for the replica group: determine whether one of thereplicas added to the failover quorum stores data or metadata that wasmore recently updated than data or metadata stored by the one of thereplicas; and in response to determining that a given replica of thereplicas added to the failover quorum stores data or metadata that wasmore recently updated than data or metadata stored by the one of thereplicas, attempting to synchronize the data or metadata stored by theone of the replicas with the more recently updated data or metadatastored by the given replica.
 5. The system of claim 4, wherein the oneof the replicas is further configured to, prior to assuming the role ofmaster replica for the replica group: in response to successfullysynchronizing the data or metadata stored by the one of the replicaswith the more recently updated data or metadata stored by the givenreplica: propagate the more recently updated data or metadata to one ormore other replicas in the replica group; propagate metadata indicatingan impending change of mastership for the replica group to the otherreplicas of the replica group; determine whether the propagation of themetadata indicating an impending change of mastership meets apre-determined durability requirement in the replica group; and inresponse to determining that the propagation of the metadata indicatingan impending change of mastership meets the pre-determined durabilityrequirement in the replica group, commit the more recently updated dataor metadata in the system.
 6. A method, comprising: performing by acomputer: one of two or more replicas that make up a replica groupattempting to assume a role of master replica for the replica group,wherein the replica group maintains an indicator of its membershipversion, wherein the membership version is updated each time themembership of the replica group changes, wherein the two or morereplicas store data on respective computing nodes of a plurality ofcomputing nodes that collectively implement a data store, wherein atmost one of the replicas in the replica group can act as a masterreplica for the replica group at a time, and wherein replicas in thereplica group that are not acting as the master replica act as slavereplicas in the replica group; wherein attempting to assume the role ofmaster replica comprises: indicating an intention to assume the role ofmaster replica; receiving credentials authorizing the attempt to assumethe role of master replica, wherein when generated, the credentials arethe highest credentials existing in the data store; gathering stateinformation from at least some of the two or more replicas other thanthe one of the replicas; for each of the at least some of the two ormore replicas: determining, dependent on the state information, whetherthe replica supports the attempt to assume the role of master replica,wherein said determining is further dependent on the receivedcredentials; and in response to determining that the replica supportsthe attempt to assume the role of master replica, including the replicain a failover quorum; and the one of the replicas assuming the role ofmaster replica for the replica group in response to determining that atleast a pre-determined number of replicas are included in the failoverquorum.
 7. The method of claim 6, wherein determining whether a replicasupports an attempt to assume the role of master replica comprises oneor more of: determining whether the replica has observed a more recentmembership version for the replicas storing the data than a most recentmembership version observed by the one of the replicas attempting toassume the role of master replica, wherein the membership version isincremented each time a membership change is made in the replica group;determining whether the replica is no longer hosted on the computingnode from which the state information for the replica is gathered; ordetermining whether the replica has seen a more recent value for a lockassociated with the replica group than a most recent lock value observedby the one of the replicas attempting to assume the role of masterreplica, wherein the most recent lock value observed by the one of thereplicas attempting to assume the role of master replica is included inthe received credentials.
 8. The method of claim 6, wherein saidattempting to assume the role of master for the replica group isperformed in response to a failure of a master replica, a failure of acomputing node on which a master replica is hosted, a communicationfailure between a master replica and one or more other components of thedata store, or a membership change in the replica group.
 9. The methodof claim 6, wherein the data store stores data in a plurality of tables;wherein, for each table, the data store maintains a plurality ofreplicas of table data in each of one or more partitions of the table onrespective computing nodes; and wherein said maintaining the datacomprises maintaining two or more replicas of the data stored in a givenpartition of a table on respective computing nodes.
 10. The method ofclaim 9, wherein said indicating an intention to assume the role ofmaster replica comprises expressing an interest in acquiring a lockassociated with the replica group; and wherein the lock associated withthe replica group is identified by an identifier of the given partition.11. The method of claim 6, wherein said indicating an intention toassume the role of master replica comprises expressing an interest inacquiring a lock associated with the replica group to an external lockservice or lock manager; wherein the credentials are received from theexternal lock service or lock manager; wherein the method furthercomprises acquiring the lock associated with the replica group; andwherein only one of the two or more replicas can hold the lock at atime.
 12. The method of claim 6, wherein said indicating an intention toassume the role of master replica comprises expressing an interest inacquiring a lock associated with the replica group; wherein the methodfurther comprises acquiring the lock associated with the replica group;and wherein a lock value for the lock is incremented each time the lockis acquired by a replica.
 13. The method of claim 6, wherein thepre-determined number of replicas is expressed in terms of a number ofreplicas stored on computing nodes in each of a particular number ofdifferent locations.
 14. The method of claim 6, wherein said gatheringcomprises gathering state information until: the pre-determined numberof replicas has been added to the failover quorum, state information hasbeen gathered from all of the two or more replicas, it is determinedthat there are not enough replicas supporting the attempt to be able toadd the pre-determined number of replicas to the failover quorum, or apre-determined time limit is reached.
 15. The method of claim 6, furthercomprising, subsequent to the one of the replicas assuming the role ofmaster replica for the replica group: another replica attempting toassume the role of master replica for the replica group; and the otherreplica abandoning its attempt to assume the role of master replica forthe replica group in response to determining that fewer than thepre-determined number of replicas are included in its failover quorum.16. The method of claim 6, further comprising, prior to the one of thereplicas assuming the role of master replica for the replica group: theone of the replicas determining whether one of the replicas added to thefailover quorum stores data or metadata that was more recently updatedthan data or metadata stored by the one of the replicas; and in responseto determining that a given replica of the replicas added to thefailover quorum stores data or metadata that was more recently updatedthan data or metadata stored by the one of the replicas, the one of thereplicas attempting to synchronize the data or metadata stored by theone of the replicas with the more recently updated data or metadatastored by the given replica.
 17. The method of claim 16, furthercomprising, prior to the one of the replicas assuming the role of masterreplica for the replica group: in response to successfully synchronizingthe data or metadata stored by the one of the replicas with the morerecently updated data or metadata stored by the given replica, the oneof the replicas propagating the more recently updated data or metadatato one or more other replicas in the replica group.
 18. The method ofclaim 17, further comprising, prior to the one of the replicas assumingthe role of master replica for the replica group, the one of thereplicas: propagating metadata indicating an impending change ofmastership for the replica group to the other replicas of the replicagroup; determining whether the propagation of the metadata indicating animpending change of mastership meets a pre-determined durabilityrequirement in the replica group; and in response to determining thatthe propagation of the metadata indicating an impending change ofmastership meets the pre-determined durability requirement in thereplica group, committing the more recently updated data or metadata inthe data store.
 19. The method of claim 6, wherein the data storemaintains data on behalf of one or more storage service clients.
 20. Anon-transitory, computer-readable storage medium storing programinstructions that when executed on one or more computers cause the oneor more computers to perform: maintaining data in a distributed datastore, wherein maintaining the data comprises maintaining two or morereplicas of the data stored on respective computing nodes, wherein thetwo or more replicas make up a replica group, wherein the replica groupmaintains an indicator of its membership version, wherein the membershipversion is updated each time the membership of the replica groupchanges, wherein at most one of the replicas in the replica group canact as a master replica for the replica group at a time, and whereinreplicas in the replica group that are not acting as the master replicaact as slave replicas in the replica group; one of the two or morereplicas attempting to assume a role of master replica for the replicagroup, wherein said attempting to assume the role of master replicacomprises: gathering state information from at least some of the two ormore replicas other than the one of the replicas; for each of the atleast some of the two or more replicas: determining, dependent on thestate information, whether the replica supports the attempt to assumethe role of master replica; and in response to determining that thereplica supports the attempt to assume the role of master replica,including the replica in a failover quorum; in response to determiningthat at least a pre-determined number of replicas are included in thefailover quorum, the one of the replicas: propagating metadataindicating an impending change of mastership for the replica group tothe other replicas of the replica group; determining whether thepropagation of the metadata indicating an impending change of mastershipmeets a pre-determined durability requirement in the replica group; andin response to determining that the propagation of the metadataindicating an impending change of mastership meets the pre-determineddurability requirement in the replica group, assuming the role of masterreplica for the replica group. 21-35. (canceled)